How Do Diffusion LLMs Generate Text Faster?

TL;DR

Diffusion language models generate a rough version of the entire response simultaneously, then refine it through repeated denoising steps instead of predicting one token after another. Inception Labs claims its Mercury coding models exceed 1,000 tokens per second on a standard Nvidia H100, potentially reducing the time and cost of coding, reasoning, tool use, and agentic workflows.

Transcript

a breakthrough in large language models just happened and it claims to be 10 times faster and 10 times less expensive using a completely novel technique lifted from text to image generation models this is diffusion large language models the way that traditional large language models work is it generates one token and then it generates the next toke... Read More

Key Insights

• Diffusion language models generate a rough approximation of the complete response at once, then repeatedly refine that output. This differs from autoregressive models, which must produce every token sequentially because each new token depends on the tokens that came before it.
• The text-generation process resembles diffusion image generation, where a noisy image is progressively denoised as a whole instead of being created one pixel at a time. Inception Labs applies this coarse-to-fine refinement process to complete blocks of text and code.
• Inception Labs describes Mercury as the first production-grade diffusion-based language model. The company claims it can run at more than 1,000 tokens per second on a standard Nvidia H100 without requiring custom hardware, making its reported speed an architectural result.
• Mercury is claimed to be 10 times faster and 10 times less expensive than competing approaches. The transcript presents one comparison in which Mercury finishes a coding task in 6 seconds, while ChatGPT and Claude require 36 seconds and 28 seconds, respectively.
• Faster inference can expand practical test-time computation because a model can perform more reasoning and error correction without making users wait for minutes. The presenter argues that this could compress lengthy coding-agent workflows, sometimes described as taking 5 to 15 minutes, into much shorter interactions.
• Diffusion models can consider and revise the entire output rather than being restricted to previously generated text. Inception Labs argues that this property may improve reasoning, response structure, mistake correction, and hallucination correction because every refinement pass can modify multiple parts of the answer.
• Mercury specializes in code generation, and the demonstrations include an HTML5 Canvas particle system, a simple Python model, and a console snake game. The presenter notes that these tasks are simple, so the demonstrations primarily establish speed and basic functionality rather than performance on complex software projects.
• Controllable generation is enabled by the model's ability to edit output and generate tokens in any order. The transcript says this could support text infilling, alignment with objectives such as safety, and more reliable compliance with user-specified output formats, alongside RAG, tools, and agentic workflows.

Questions & Answers

Q: How do diffusion language models generate text?

Diffusion language models begin with a rough, noisy approximation of the entire response and improve it through repeated denoising steps. They do not have to finalize one token before starting the next token. The process is compared with diffusion image generation, which transforms noise into a recognizable image by refining the whole image over time rather than drawing one pixel at a time.

Q: How are diffusion LLMs different from autoregressive LLMs?

Autoregressive language models generate one token and then use that token, together with all preceding text, to generate the next one. This creates an inherently sequential process. Diffusion language models instead create an initial approximation of the complete output and iteratively revise it. Because they can work across the response during each refinement step, they are not limited to considering only previously finalized output.

Q: How fast is Inception Labs' Mercury model?

Inception Labs claims that Mercury runs at more than 1,000 tokens per second on an Nvidia H100, a standard chip also used by other language models. The transcript describes it as 10 times faster than frontier speed-optimized language models. In one presented comparison, Mercury completes a coding task in 6 seconds, while ChatGPT takes 36 seconds and Claude takes 28 seconds.

Q: Why could diffusion LLMs improve test-time reasoning?

Faster inference allows a model to perform more computation during generation while keeping the waiting time practical. The transcript says current reasoning models may think for minutes when operating at 40, 50, or 60 tokens per second. At more than 1,000 tokens per second, substantially more reasoning and error-correction work could be completed in a few seconds or within the same allotted time.

Q: Can diffusion language models correct mistakes in their output?

Inception Labs argues that diffusion models can correct mistakes and hallucinations because they repeatedly refine a complete draft. Unlike an autoregressive model that commits to an ordered sequence of tokens, a diffusion model can reconsider different portions of its output during successive passes. The claimed benefit is improved reasoning and structure, although the transcript presents this as the company's explanation rather than an independently demonstrated conclusion.

Q: What coding tasks does Mercury demonstrate?

The demonstrations show Mercury creating an HTML5 Canvas particle system whose particles follow the mouse, with controls for speed, size, and color. It also writes a simple Python model and produces a snake game that runs in the console. The presenter explicitly characterizes the examples as simple, using them mainly to illustrate the model's rapid generation and functional code output.

Q: How could diffusion LLMs make AI agents faster?

AI agents often generate many intermediate outputs while planning, reasoning, using tools, and iterating on solutions. The transcript identifies model speed as a major bottleneck, especially in coding workflows where a user might wait 5, 10, or 15 minutes. A much faster model could complete more agent steps in the same period, potentially improving both productivity and the amount of reasoning performed.

Q: What applications can diffusion language models support?

The transcript says diffusion language models support RAG, tool use, and agentic workflows. It also identifies advanced reasoning, controllable generation, and edge applications as promising uses. Because the model can revise tokens in any order, it may fill missing text, align output with objectives such as safety, and conform more reliably to requested formats. Smaller capable models could also run on laptops or desktops.

Summary & Key Takeaways

• Traditional autoregressive language models generate tokens sequentially because every new token depends on preceding output. Inception Labs instead applies a diffusion process to text, beginning with a noisy approximation of the complete response and refining it over several passes until the text becomes coherent, structured, and suitable as a final answer.

• Mercury is presented as the first production-grade diffusion language model and as a code-generation specialist. Demonstrations include creating an HTML5 Canvas particle system, writing a simple Python model, and producing a console snake game. The examples emphasize rapid generation, although the presenter acknowledges that the demonstrated programming tasks are relatively simple.

• The architecture could accelerate agents, test-time computation, controllable generation, and edge applications. Faster inference allows more reasoning and correction within a practical waiting period. Because diffusion models can revise any part of an output, Inception Labs argues that they can improve structure, correct mistakes, fill text, and follow requested formats more reliably.