How Do Attention Residuals Improve LLM Depth?

TL;DR

Attention residuals let each layer selectively access earlier layer outputs with trainable weights, preserving information across depth and avoiding repeated compression. The block attention variant reduces memory and compute while nearly matching full attention residuals, providing scalable, efficient depth in transformers.

Transcript

In the first three months of 2026, we haven't already seen some of the biggest LM architectural breakthrough in a long time. While a good chunk of 2025 was playing around with RL to apply it onto LMS, which got us more and more acronyms that I don't even know what they stand for, I am so glad that my videos this year is not going to be spending 10 ... Read More

Key Insights

• Attention residuals create a direct, trainable connection from every previous layer to the current layer, enabling selective retrieval of past representations rather than fixed, single-path depth accumulation.
• The method replaces a fixed additive shortcut with a dynamic, input dependent routing mechanism across depth, so later layers can amplify or ignore specific earlier signals based on current input.
• Quadratic scaling is a challenge if every layer attends to every other layer, so the authors propose block attention residuals to group layers into blocks and summarize within blocks before cross-block attention.
• Block attention residuals reduce memory and compute from total layer count to total block count, trading some granularity for substantial efficiency gains without large losses in performance.
• Empirical results show block attention residuals perform close to full attention residuals across scales and can match baseline performance with about 25 percent lower compute.
• The approach improves information preservation by keeping earlier layer signals accessible, allowing useful signals to propagate without being drowned by repeated mixing.
• The method also fixes the magnitude problem in standard residual setups, as attention based weighting favors relevance rather than simply larger outputs from later layers.
• Depth guided by attention allows the model to reuse prior computations, reducing information loss and improving training efficiency across deep architectures.
• More videos from this channel:
  • How do 1-bit LLMs achieve maximum efficiency?
  • How RL and LoRA reshape efficient LLMs
• More from this channel: bycloud

Questions & Answers

Q: What is attention residuals in simple terms and what problem do they try to solve?

Attention residuals are a mechanism where each current layer in a transformer can directly access outputs from all previous layers using attention to decide how much to rely on each one. This solves the problem of information being diluted as depth increases, where repeated compression of earlier signals can cause loss of important nuances and lead to inefficiencies in training.

Q: How does the attention residual concept differ from traditional residual connections?

Traditional residual connections add a fixed, local shortcut from input to output within a layer, preserving information but not enabling access to the full history of past layers. Attention residuals extend this idea across depth, allowing the current layer to attend to every earlier layer with learned weights, providing a dynamic, input dependent routing of signals through depth.

Q: What is block attention residuals and why were they proposed?

Block attention residuals group layers into blocks and allow normal attention within a block, but at the end of the block the outputs are summarized. Across blocks, the model only attends to these summaries rather than every individual layer. This reduces memory and compute dramatically while keeping performance close to full attention residuals, enabling scalable models.

Q: How do block attention residuals affect compute and performance according to the paper?

The paper shows that block attention residuals can match the performance of full attention residuals while using about 25 percent less compute, with only a small overhead. This means training can be faster and cheaper, while maintaining similar loss improvements across scales.

Q: What is the main advantage of attention over depth compared to attention over tokens?

The main advantage is preserving earlier information by giving each layer access to multiple past representations rather than compressing information via repeated mixing. This allows the model to retrieve relevant historical signals when needed, improving information preservation and learning efficiency.

Q: What drawback does attention over depth still face and how is it mitigated?

A key drawback is quadratic scaling, since each layer could attend to many previous layers. The mitigation is block attention residuals, which reduce the number of attention targets by organizing layers into blocks and attending to block summaries rather than every layer.

Q: How does the residual path change with attention residuals compared to standard residuals?

In attention residuals, the residual path becomes a trainable, input dependent routing mechanism. Instead of simply adding inputs, the model forms a query against all previous layer outputs and uses normalized scores as weights, producing a weighted combination that reflects relevance to the current input.

Q: What qualitative benefits do the researchers claim for attention residuals beyond compute savings?

Beyond compute savings, attention residuals improve information preservation by keeping earlier layers accessible and reducing magnitude competition among layers. This leads to more stable training and better utilization of signals across depth, enabling deeper models without the typical degradation from depth.

Summary & Key Takeaways

• The video explains a new concept called attention residuals, which enables layers in a transformer to attend to all previous layer outputs instead of only the immediate input, preserving information and stabilizing training.

• A practical improvement, block attention residuals group layers into blocks to reduce compute and memory without greatly sacrificing performance, making deep architectures more feasible at scale.

• The concept is motivated by problems with depth in transformers, where information can get diluted and later layers must produce larger outputs to influence the final representation, and shows how attention routing fixes this with efficiency.