Beyond Softmax: The Future of Attention Mechanisms

Transcript

Attention is the core building block for generative AI. It allows us to capture contextual information within a sequence. But standard attention mechanisms still suffer from quadratic compute and linear memory constraints. Can we overcome these limitations? Let's first review how standard attention works. The input to our attention layer is a seque... Read More