How Does DiffusionLM Generate Text Faster Than Standard LLMs?

TL;DR

DiffusionLM generates text faster by predicting multiple masked token positions in parallel instead of producing tokens sequentially from left to right. The method begins with a fully masked sequence and repeatedly denoises it using a transformer until the sequence is unmasked. Mercury Coder Mini and Gemini Diffusion illustrate the approach’s commercial progress, while questions about quality and broader capabilities remain. Read on for the mechanics, comparisons, and limitations.

Transcript

AI based applications currently have a major problem that is an insanely long wait time. Not only does ALM now think which can take a while to get an answer, but when incorporated with other AI agentic applications, the weight is even more painful. And the only few solutions we currently have right now is either use a dumper model or just have bett... Read More

Key Insights

• Diffusion language models generate text by starting with a fully masked sequence and denoising discrete tokens across multiple iterations. At each step, the model predicts a subset of masked positions in parallel, using a masking schedule or confidence threshold until the sequence becomes completely unmasked.
• The underlying architecture of a diffusion language model can still be a transformer. Diffusion describes the generation method rather than requiring an entirely different architecture, with the transformer repeatedly processing the masked context and implementing a coarse-to-fine denoising procedure.
• Parallel token prediction is the main source of the reported speed advantage. Mercury has been measured at around 800 tokens per second on an H100 GPU, while Gemini Diffusion reportedly peaks at nearly 1,500 tokens per second with a 32K context window.
• Mercury Coder Small is a coding-focused diffusion model with a 32K context window. Its listed API pricing is 25 cents per million input tokens and $1 per million output tokens, and it is the commercial diffusion model in the comparison that already offers API access.
• Mercury Coder performs especially well on fill-in-the-middle coding tasks. Its official benchmarks show accuracy improvements of 22% to 30% over the compared competitors, although the transcript notes that its broader results have little third-party validation beyond the Copilot Arena.
• Copilot Arena ranks Mercury sixth among the models included in its autocomplete evaluation. That position requires context because the arena contains non-reasoning models, many models are around a year old, and completion speed does not influence the displayed A/B comparison results.
• Gemini Diffusion delivers similar coding benchmark performance to Mercury while roughly doubling its raw generation speed. Its reported peak is nearly 1,500 tokens per second, but Mercury retains a practical advantage because it is already accessible through an API.
• Gemini Diffusion currently trails Gemini 2.0 Flash-Lite on several non-coding evaluations. Its official results show 16% lower GPQA accuracy, 10% lower multilingual question-answering performance, and 6% lower reasoning performance, alongside a 3% improvement on AIME mathematics.

Questions & Answers

Q: How do diffusion language models generate text faster than standard LLMs?

Diffusion language models predict a subset of masked token positions in parallel during each denoising step. Standard autoregressive models instead generate tokens sequentially from left to right, making parallel prediction the main source of the diffusion method’s speed advantage.

Q: How does a diffusion language model generate text?

It starts with a fully masked sequence and denoises that sequence over multiple iterations. At each step, it predicts and replaces some masked positions according to a masking schedule or confidence threshold until the sequence is completely unmasked.

Q: Do diffusion language models still use transformers?

Yes. Diffusion describes the generation method, while the underlying architecture can still be a transformer that repeatedly processes the entire masked context.

Q: Why can diffusion language models use bidirectional or global context?

They process the entire masked sequence rather than being restricted to tokens already generated on the left. The transcript says this bidirectional or global context can make the approach useful for structured or logical tasks.

Q: What is Mercury Coder Mini?

Mercury Coder Mini is a commercially available diffusion language model from Inception Labs focused on code generation. The transcript presents it as substantially faster than the small autoregressive coding models used in its comparison.

Q: What is Gemini Diffusion?

Gemini Diffusion is Google DeepMind’s diffusion language model. The transcript describes it as available through a waitlist and demonstrates generation that appears even faster than Mercury Coder Mini.

Q: How does Gemini Diffusion compare with Mercury Coder?

Both use diffusion-based text generation and demonstrate very fast output. Gemini Diffusion is presented as having greater raw generation speed, while Mercury Coder has the practical advantage of commercial availability and API access.

Q: What are the current limitations of diffusion language models?

The existing fields report that current diffusion models can still trail autoregressive models on several language and reasoning evaluations. They also identify fixed context windows, reasoning-data requirements, denoising choices, model size, and cache optimization as unresolved challenges.

Summary & Key Takeaways

• Diffusion language models begin with a fully masked sequence and progressively replace masked positions over multiple denoising iterations. A transformer processes the entire masked context, allowing several tokens to be predicted in parallel. This coarse-to-fine method differs from the strictly left-to-right generation used by conventional autoregressive language models.

• Mercury Coder demonstrates the practical speed and coding strengths of commercial diffusion models. Reported measurements place Mercury near 800 tokens per second on an H100 GPU, while its fill-in-the-middle accuracy exceeds compared models by 22% to 30%. However, limited independent benchmarking makes its broader competitive position difficult to establish.

• Gemini Diffusion reportedly reaches nearly 1,500 tokens per second and uses a 32K context window, but its official results reveal uneven quality outside coding. Compared with Gemini 2.0 Flash-Lite, it falls behind on GPQA, multilingual question answering, and reasoning, while recording a small advantage on the AIME mathematics benchmark.