Andrew Feldman on How Cerebras Plans to Beat Nvidia GPUs

TL;DR

Generative AI inference is limited by memory bandwidth, not calculation. Every new word requires moving all model weights from memory to compute. Andrew Feldman argues that the GPU's slow, high-capacity off-chip HBM memory is a structural weakness for inference. Cerebras builds wafer-scale chips packed with fast on-chip SRAM. It handles manufacturing flaws by building the processor from redundant identical tiles.

Transcript

our AI algorithms today are not particularly efficient in a GPU most of the time it's doing inference it's 5 or 7% utiliz that means it's 95 or 93% wasted we won't be as dependent on Transformers in 3 years or 5 years as we are now 100% the fundamental architecture of the GPU with off chip memory is not great for inference now they will continue to... Read More

Key Insights

• AI computation is simple at its core: the underlying calculation is matrix multiplication, and Feldman says a second-year electrical engineering student could design the multiply-accumulate unit. The hard part is moving results and intermediate results to and from memory and between chips.
• Generative inference is limited by memory bandwidth because all model weights must move from memory to compute to generate a single word, and again for every following word. A 70 billion parameter model at 16 bits per weight moves about 140 gigabytes of data per word.
• GPUs are poorly utilized during inference: Feldman says a GPU doing inference is often only 5 to 7 percent utilized, which means 93 to 95 percent of its capacity is wasted. He describes current AI algorithms as not particularly efficient on GPUs.
• HBM and SRAM make opposite tradeoffs: HBM, the memory GPUs use, is high capacity but slow, which suited graphics workloads that rarely went back to memory. SRAM is extremely fast but low capacity, so a normal-sized SRAM chip cannot hold a large model.
• Wafer-scale design is how Cerebras gets both SRAM speed and enough capacity. With normal-sized SRAM chips, Feldman estimates a 400 billion parameter model would need about 4,000 chips, and DeepSeek 671 would need six to eight thousand.
• Nvidia's original strength has turned into a weakness for inference, according to Feldman. Building GPUs around off-chip memory gave them an advantage over CPUs, but dedicated AI chips now outperform them on inference. Nvidia also depends on memory makers such as SK Hynix, Samsung, and Micron.
• Wafer-scale chips use less power because moving data off chip through I/O is among the most power-hungry operations a chip performs. Keeping data inside the silicon on a single wafer avoids much of that energy cost.
• Cerebras solved the yield problem by building its processor from hundreds of thousands of identical tiles plus spare rows or columns. When a flaw appears, the bad tile is shut down and a spare is used in its place, a technique borrowed from memory manufacturing.

Questions & Answers

Q: Why is memory bandwidth the bottleneck for AI inference?

In generative inference, every model weight must be moved from memory to the compute units to generate a single word, and then moved again for each word after it. For a 70 billion parameter model with 16-bit weights, that means moving about 140 gigabytes of data per word. The memory bandwidth needed for this repeated movement is the fundamental limit on inference speed in architectures like the GPU.

Q: What is the difference between HBM and SRAM for AI chips?

HBM is the memory GPUs use. It has high capacity but is slow, which was fine for graphics, where the processor rarely had to go back to memory. SRAM is extremely fast but has low capacity, so a normal-sized chip built with SRAM cannot hold a large model. Cerebras chose SRAM and went to wafer scale to fit enough of it to get both speed and capacity.

Q: Why does Cerebras build chips at wafer scale?

Wafer scale lets Cerebras put a very large amount of fast SRAM on one piece of silicon. With normal-sized SRAM chips, Feldman estimates a 400 billion parameter model would need about 4,000 chips, and DeepSeek 671 would need six to eight thousand, which he calls an administrative nightmare. A wafer keeps the speed of SRAM while providing the capacity to hold models on one or a few wafers.

Q: How efficient are GPUs at running AI inference?

Feldman says a GPU running inference is usually only about 5 to 7 percent utilized, which means 93 to 95 percent of its capacity is wasted. He argues that current AI algorithms are not particularly efficient on GPUs. He also says the GPU's architecture of off-chip memory is not well suited to inference: Nvidia will keep doing well in inference, but it can be beaten.

Q: Why has Nvidia's GPU strength become a weakness?

GPUs were built for graphics, around off-chip, high-capacity memory, and that design gave them an advantage over CPUs. Now that dedicated AI chips exist, Feldman says, that former advantage has become a weakness for inference. Nvidia also does not make its own memory. It buys from a small number of suppliers such as SK Hynix, Samsung, and Micron, which limits its options.

Q: What is chip yield and why is it hard at wafer scale?

A wafer is a 12-inch silicon disc, and chips are cut out of it like cookies from dough. Manufacturing flaws land at random, like M&Ms tossed onto the dough, so a bigger chip is more likely to contain a flaw. Traditionally a flawed chip was thrown away or sold as a cheaper part through binning. That is why people believed a wafer-sized chip could never yield well.

Q: How did Cerebras solve the yield problem for wafer-scale chips?

Cerebras builds its processor from hundreds of thousands of identical tiles and adds redundant rows or columns of spare tiles. When a tile has a flaw, it is shut down, the system routes around it, and a spare tile takes its place. The technique came from memory manufacturing, where yields are extremely high. Feldman says Cerebras now yields as well as or better than companies building much smaller chips.

Q: Can one chip architecture handle training, fine-tuning, and inference?

Feldman says the computational work of training from scratch and of fine-tuning is approximately the same, so one architecture handles both. Inference, especially generative inference, places heavier demands on communication and memory bandwidth. Cerebras chose to target all three, and Feldman notes that the first step in computer architecture is deciding what you will not be good at.

Summary & Key Takeaways

• Cerebras was founded in 2015 after its co-founders saw AI emerging as a new computing workload that would strain memory bandwidth and communication rather than raw calculation. Feldman, on his fifth startup, says he badly underestimated the market's size but correctly predicted that AI would put a different kind of pressure on processors.

• Feldman explains that generative inference has to move every model weight from memory for each word it produces, so memory bandwidth becomes the fundamental bottleneck. GPUs use slow, high-capacity HBM. Cerebras uses wafer-scale chips full of fast SRAM, and says it has been the fastest inference provider since launching on August 26th.

• Wafer-scale design required solving yield, the problem that bigger chips are more likely to contain manufacturing flaws. Cerebras builds processors from hundreds of thousands of identical tiles with redundant spares, routes around defective tiles, and says it yields as well as or better than makers of much smaller chips, while also using less power.