How Are GPT, Claude, and Gemini Actually Trained and Served?, Reiner Pope

TL;DR

GPT, Claude, and Gemini are trained and served by distributing model computation and memory work across clusters while balancing batch size, bandwidth, latency, and cost. Reiner Pope analyzes inference on a Blackwell NVL72 rack of 72 GPUs and explains why batching users can improve serving economics dramatically. Read on to understand how model architecture, KV caches, parallelism, and API pricing fit together.

Transcript

Today, I'm interviewing Reiner Pope, who is the CEO of MatX, which is a new chip startup. Previously, he was doing TPU architecture and many other things at Google. This is a very different format from my usual interviews. This is going to be a blackboard lecture. We're going to get up in a second. We in fact built this whole new studio with specif... Read More

Key Insights

• Batch size significantly impacts token cost and speed, with larger batch sizes reducing cost per token by amortizing memory fetches.
• Model architecture, such as mixture of experts (MoE), influences how models are laid out across GPU racks, optimizing for communication patterns.
• Pipeline parallelism spreads model layers across racks, reducing memory capacity requirements but not necessarily improving latency.
• Inference often uses expert parallelism within a single scale-up domain, minimizing pipelining due to memory bandwidth constraints.
• The balance between training, RL generation, and inference costs can guide optimal model over-training beyond Chinchilla scaling.
• API pricing can reveal underlying costs, such as the impact of context length on memory bandwidth and compute time.
• Cache hits are significantly cheaper due to reduced rematerialization costs, highlighting the importance of efficient memory management.
• Neural networks and cryptographic protocols share structural similarities, but aim for opposite goals: extracting structure vs. creating randomness.

Questions & Answers

Q: How are GPT, Claude, and Gemini trained and served?

Their models are run across clusters of chips, with architecture and infrastructure determining how computation, weights, and KV caches are distributed. Training and inference use strategies such as expert parallelism and pipeline parallelism while balancing compute throughput, memory bandwidth, latency, and cost.

Q: Why can AI services charge more for faster token generation?

Reiner Pope identifies batch size as the main effect: serving fewer users together can provide faster latency but makes each token more expensive. The transcript cites Fast Mode offerings that cost 6x as much while streaming tokens at 2.5x the speed.

Q: How does batch size affect AI inference cost and speed?

A batch combines many users so the system can process them at the same time and amortize weight-fetch costs across more tokens. Pope says serving without batching can produce economics a thousand times worse than batching many users, although larger batches involve a latency trade-off.

Q: What does roofline analysis reveal about transformer inference?

Roofline analysis estimates inference time by examining compute performance and memory bandwidth. Pope applies it to a Blackwell NVL72 cluster, a rack of 72 GPUs, and separates the work into operations on model weights and operations on the context stored in the KV cache.

Q: What determines the compute time for generating an AI token?

The estimate considers the batch size, the model’s number of active parameters, and the chips’ compute throughput. It focuses on multiplying by the active parameters and treats attention computation as comparatively small for this approximation.

Q: Why do active and total parameter counts both matter?

Active parameters determine the weight-matrix computation required for one token, while total parameters affect how many weights must be fetched from memory. The transcript’s example, DeepSeek V3, has about 37 billion active parameters and 700 billion total parameters.

Q: What is the KV cache, and why does it affect inference?

The KV cache represents previously processed context used during autoregressive decoding. Its memory-fetch work grows with batch size because the system must fetch a context-length quantity of data for every batch element, multiplied by the model’s bytes per token.

Q: How do model architecture and parallelism affect training and inference?

Mixture-of-experts architecture influences how model components are arranged across GPU racks and can use expert parallelism for communication within a scale-up domain. Pipeline parallelism spreads layers across racks to reduce per-rack memory capacity requirements, but it does not inherently improve latency and is more useful in training than inference.

Summary & Key Takeaways

• Batch size plays a crucial role in determining token cost and speed, with larger batch sizes reducing costs by better utilizing memory bandwidth. Understanding the interaction between batch size and latency is key to optimizing AI models.

• Model architecture, such as mixture of experts, affects how models are distributed across GPU racks. Expert parallelism and pipeline parallelism are strategies used to optimize model training and inference.

• API pricing provides insights into the costs associated with context length and cache management. Longer contexts are more expensive due to increased memory bandwidth demands, while cache hits are cheaper, highlighting the importance of efficient memory usage.