How Etched Is Building AI Inference Hardware

TL;DR

Etched is building a full rack-scale inference system, not merely an AI chip, because performance depends on compute, memory, interconnects, power delivery, boards, and production working together. Its founders argue that specializing for inference, separating prefill from decode, and solving thermal limits through low-voltage design can deliver substantially higher useful throughput than general-purpose hardware.

Transcript

We know inference is going to be the biggest market in the world. Whoever produces the most tokens is going to be the most valuable company in the world. >> We we had people quit. >> Yeah. >> That people literally were like, "This problem is unsolvable." And uh best of luck, guys. >> You kind of have to be sick in the head to join our company. You'... Read More

Key Insights

• Etched is building a complete inference rack rather than an isolated processor. Its product encompasses the chip, circuit board, power delivery, chip-to-chip interconnect, rack design, and manufacturing process because each system component affects practical performance at mass-production scale.
• Inference consists of two major stages called prefill and decode. Prefill reads a large volume of existing text and prepares the model's KV cache, while decode uses that cached model state to generate successive output tokens.
• Prefill-decode disaggregation assigns the two inference stages to different clusters. After a prefill cluster prepares the relevant KV cache, the system transfers that model memory to a decode cluster, which uses it to produce the next tokens.
• Specialization can remove constraints inherited from general-purpose semiconductor tools and components. The founders cite default timing assumptions about operation at freezing temperatures, even though they expect AI data-center chips to operate at temperatures no lower than roughly 80 degrees Celsius.
• Small improvements across a system can compound into a much larger inference advantage. Etched seeks gains across design tools, circuits, power modules, boards, chips, and interconnects instead of expecting one isolated architectural change to produce every performance improvement.
• Model FLOPS utilization measures how much advertised peak computation is actually useful on real workloads. The founders state that GPUs often achieve between 20 and 50 percent utilization, depending on the workload, making headline peak FLOPS an incomplete performance measure.
• Thermal limits can prevent additional compute units from increasing real performance. As more transistors switch, power consumption and heat rise, causing a chip to regulate itself by lowering its clock speed, so Etched treats heat management as a prerequisite for higher useful throughput.
• Truth-seeking skepticism helped Etched recruit experienced supporters. Semiconductor expert Mark Ross initially rejected the founders' proposal, then requested evidence, validated their functional simulation, became an adviser, and ultimately joined as the company's full-time chief technology officer.

Questions & Answers

Q: How is Etched designing hardware for AI inference?

Etched is designing a complete rack-scale inference solution rather than selling only a processor. The system includes the chip, its circuit board, power delivery, communication links between chips, rack-level integration, and mass production. The founders believe production itself is part of the product because useful inference performance depends on the entire system operating together.

Q: What are prefill and decode in AI inference?

Prefill and decode are the two main stages of inference described by the founders. Prefill reads the supplied text and brings the model's memory, called the KV cache, into the correct state. Decode then uses that prepared cache to generate output tokens. They compare the relationship to loading a gun and then firing it.

Q: How does prefill-decode disaggregation work?

Prefill-decode disaggregation places the two inference stages on different clusters. A prefill cluster processes the known input text and prepares the model's KV cache. That cache is then transferred to a decode cluster, which uses the stored model state to generate subsequent tokens. This structure allows hardware to be designed around each stage's distinct requirements.

Q: Why does Etched focus on the entire inference rack?

Etched focuses on the entire rack because processor performance is shaped by more than chip architecture. Power delivery, boards, interconnects, cooling-related limits, production, and communication among chips all influence system throughput. By controlling those elements together, the company can change constraints throughout the stack and combine multiple improvements into a larger overall performance gain.

Q: Why is thermal management important for inference chips?

Thermal management is important because higher compute utilization activates more transistors, increasing power consumption and heat. The founders explain that a chip can respond by lowering its clock speed to avoid overheating. Simply adding more FLOPS may therefore fail to improve real performance, which is why Etched addresses the thermal problem before increasing computational capacity.

Q: What is model FLOPS utilization in AI hardware?

Model FLOPS utilization, or MFU, measures how much of a chip's advertised peak computation is actually delivered on a real model workload. Etched's founders state that GPUs often achieve between 20 and 50 percent, depending on the workload. They therefore emphasize useful FLOPS and FLOPS density instead of relying only on a processor's headline peak figure.

Q: How did Etched overcome early skepticism?

Etched addressed serious technical skepticism by producing evidence. Mark Ross, a semiconductor expert and former Cypress Semiconductor chief technology officer, initially said the idea would not work. He asked the founders for a white paper and functional simulation. After examining their simulation, he concluded that it worked, became increasingly involved, and eventually joined full time.

Q: Why did Etched challenge standard chip-design assumptions?

Etched found that many semiconductor tools and components were designed with general-purpose constraints covering data centers, edge devices, and IoT applications. The founders questioned which constraints actually applied to inference. One example was timing analysis that assumed full-speed operation at freezing temperatures, although they expected their data-center chips to operate at temperatures of roughly 80 degrees Celsius or higher.

Summary & Key Takeaways

• Etched began with two young Harvard dropouts challenging the semiconductor industry's assumption that successful chip companies require founders with decades of experience. Early skepticism forced them to demonstrate their architecture through a white paper and functional simulation, helping turn semiconductor expert Mark Ross from a critic into the company's eventual full-time chief technology officer.

• The company designs an entire inference solution that includes chips, racks, boards, power delivery, interconnects, and mass production. Its architecture separates prefill, which prepares the model's KV cache from input text, from decode, which uses that stored context to generate tokens. The cache is transferred between specialized clusters for each stage.

• Etched evaluated multiple memory and packaging approaches before concluding that every option introduces trade-offs involving heat, supply chains, bonding, or computational capacity. Its prefill strategy emphasizes useful FLOPS density and model FLOPS utilization. Because greater utilization increases heat and can reduce clock speed, the company treats thermal efficiency as a foundational design problem.