How Does DeepSeek V4 Make Long Context Cheap?

TL;DR

DeepSeek V4 lowers long-context inference costs by combining compressed sparse attention, heavily compressed attention, and a 128-token sliding window. The design compresses historical context at different rates, retrieves only relevant mid-range blocks, and preserves recent tokens precisely, enabling a 1 million-token context window with substantially lower KV-cache use and inference compute than DeepSeek V3.2.

Transcript

After being teased for nearly two months about the Deep Seek V4 release, the Whale has finally dropped it. The fourth generation of Deep Seek, and they never cease to amaze me. Alongside the two models they released, which are DeepC V4 Pro and Flash, they also published a 58page technical report that contains nothing but gold, outlining everything ... Read More

Key Insights

• DeepSeek V4 supports a native 1 million-token context window by combining three forms of attention that serve different memory ranges. A 128-token sliding window preserves recent details, compressed sparse attention retrieves relevant historical blocks, and heavily compressed attention supplies a broad global view.
• Compressed sparse attention is a learned compression and retrieval mechanism. It combines every four tokens into one KV entry using token-generated KV information and compression weights, then employs a lightweight indexer to select only the compressed blocks that appear most relevant to each query token.
• Heavily compressed attention reduces every 128 tokens to a single KV entry. Because this representation is 32 times lighter than the four-token compression used by compressed sparse attention, the model can apply dense attention across the compressed memory to obtain an inexpensive but less precise global view.
• DeepSeek V4 interleaves compressed sparse attention and heavily compressed attention at a 1:1 ratio. Each mechanism also includes a sliding-window branch that retains the most recent 128 tokens as normal, uncompressed KV entries because recent information is especially important when predicting the next token.
• DeepSeek V4 Pro uses 27% of the single-token inference FLOPs and 10% of the KV cache used by DeepSeek V3.2 at a 1 million-token context. V4 Flash uses about 7% of V3.2's KV cache, corresponding to a reported 13.7-fold reduction.
• DeepSeek V4 Pro is a mixture-of-experts model with 1.6 trillion total parameters and 49 billion active parameters. DeepSeek V4 Flash is smaller, with 284 billion total parameters and 13 billion active parameters, while both models support contexts of up to 1 million tokens.
• The hybrid attention design is not linear attention because its computational cost still grows quadratically. Compression and sparse selection reduce the practical rate of growth, allowing the architecture to process long contexts at substantially lower cost without changing that underlying scaling classification.
• Manifold constraint hyperconnections expand the residual stream into multiple paths and learn how to mix them before and after each layer. The stated goal is to increase representational capacity across model depth and potentially help useful information survive as it passes through successive layers.

Questions & Answers

Q: How does DeepSeek V4 make long-context inference cheaper?

DeepSeek V4 makes long-context inference cheaper by combining compressed sparse attention, heavily compressed attention, and a sliding-window branch. These mechanisms respectively compress and selectively retrieve historical blocks, maintain a heavily summarized global memory, and preserve the latest 128 tokens precisely. At a 1 million-token context, V4 Pro reportedly requires 27% of V3.2's single-token inference FLOPs and 10% of its KV cache.

Q: What is compressed sparse attention in DeepSeek V4?

Compressed sparse attention, or CSA, combines every four tokens into one learned KV entry rather than retaining each token as a separate memory slot. Tokens generate candidate KV information and compression weights that determine their contribution to the summary. A lightweight indexer then scans these compressed entries and selects only the blocks that appear most relevant to the current query token.

Q: What is heavily compressed attention in DeepSeek V4?

Heavily compressed attention, or HCA, turns every 128 tokens into one KV entry. This memory is 32 times lighter than CSA's four-token compression, making dense attention over the summarized context practical. HCA is intended to provide an inexpensive global view rather than retrieve one exact historical token, so it is described as less precise than CSA but cheaper and more global.

Q: Why does DeepSeek V4 use three attention mechanisms?

DeepSeek V4 uses three attention mechanisms because recent, intermediate, and distant information have different requirements. The 128-token sliding window preserves recent tokens without compression, CSA supplies more precise retrieval from compressed historical blocks, and HCA gives the model a heavily compressed global overview. The architecture interleaves CSA and HCA at a 1:1 ratio while adding the sliding-window branch to both.

Q: How much memory does DeepSeek V4 save compared with V3.2?

At a 1 million-token context, DeepSeek V4 Pro reportedly uses about 10% of the KV cache required by V3.2, making its cache roughly 9.5 times smaller. V4 Flash uses about 7%, corresponding to a 13.7-fold reduction. Against the GQA baseline discussed in the transcript, the estimated total reductions are 34-fold for V4 Pro and 49-fold for V4 Flash.

Q: What are the model sizes of DeepSeek V4 Pro and Flash?

DeepSeek V4 Pro is described as having 1.6 trillion total parameters with 49 billion parameters active during computation. DeepSeek V4 Flash has 284 billion total parameters and activates 13 billion. Both models support context windows of up to 1 million tokens, while Flash is presented as the smaller and cheaper version of the two-model V4 series.

Q: Is DeepSeek V4 using linear attention?

DeepSeek V4 is not described as using linear attention. Its attention cost still grows quadratically, but compression and sparse selection slow that growth substantially in practical operation. CSA reduces the number of stored entries and retrieves only relevant compressed blocks, while HCA compresses memory enough to permit dense attention over a much smaller representation of the full context.

Q: What other architectural changes are used in DeepSeek V4?

DeepSeek V4 changes its residual stream through manifold constraint hyperconnections, which create multiple residual paths and learn how to mix them before and after each layer. Its feed-forward block retains DeepSeek's mixture-of-experts design with 256 fine-grained router experts and shared experts. The router activation changes from sigmoid to square-root softplus, and a sequence-level balance loss is added to reduce extreme routing imbalance.

Summary & Key Takeaways

• DeepSeek V4 addresses the cost of serving long contexts through a hybrid attention architecture. Compressed sparse attention turns every four tokens into one learned KV entry and retrieves only relevant compressed blocks, while heavily compressed attention summarizes every 128 tokens and provides an inexpensive global view through dense attention.

• The architecture interleaves compressed sparse attention and heavily compressed attention at a 1:1 ratio. Both mechanisms also receive a sliding-window branch that preserves the latest 128 tokens without compression. Together, these branches handle recent details, selective mid-range retrieval, and broad long-range information while reducing memory and computation costs.

• DeepSeek V4 Pro has 1.6 trillion total parameters and 49 billion active parameters, while V4 Flash has 284 billion total parameters and 13 billion active parameters. Both support contexts up to 1 million tokens. The models also use manifold constraint hyperconnections and an adjusted mixture-of-experts routing system.