How Do Nvidia GPUs Compare With Custom AI Chips?

TL;DR

Nvidia GPUs remain the flexible workhorses for AI training and inference, while custom ASICs from Google, Amazon, and other hyperscalers offer greater efficiency and control for specific workloads. ASICs can reduce power and computing costs at massive scale, but they require tens or hundreds of millions of dollars to develop and cannot be changed after being carved into silicon.

Transcript

Nvidia graphics processing units like these latest Blackwell GPUs, are inside server racks all over the world. Nvidia has catapulted from gaming giant, to the very core of generative AI, training the models, running the workloads and sending Nvidia's valuation soaring. With 6 million Blackwell GPUs shipped over the last year. This connects a... Read More

Key Insights

• GPUs are parallel processors with thousands of smaller cores that efficiently perform matrix multiplication and other calculations across multidimensional data structures called tensors. That architecture supports both major phases of AI computation: training models on large datasets and running inference on new information.
• Nvidia sells complete AI systems in addition to individual GPUs. A 72-GPU Blackwell server rack connects its processors so they can act as a single GPU, costs around $3 million, and was being shipped at a stated rate of 1,000 systems each week.
• AMD is Nvidia's leading GPU competitor and uses a largely open-source software ecosystem for its Instinct products. Nvidia GPUs are tightly optimized around CUDA, Nvidia's proprietary software platform, while both companies are pursuing rack-scale systems and commitments from major AI and cloud customers.
• Custom ASICs are hard-wired processors designed to execute specific types of AI mathematics with high speed and efficiency. Their specialization can lower power and computing costs, but the silicon cannot be changed after production, creating a fundamental tradeoff between efficiency and workload flexibility.
• Custom AI chips require enormous upfront investment, with development costing at least tens and often hundreds of millions of dollars. Startups therefore tend to rely on commercially available GPUs, while hyperscalers can justify ASIC programs through scale, infrastructure control, power efficiency, and reduced Nvidia dependence.
• Google's TPU was the first custom ASIC for AI acceleration, debuting in 2015. Google released the seventh-generation Ironwood a decade later and reached an agreement for Anthropic to train Claude using up to one million TPUs, although TPUs have traditionally served Google's internal environment.
• AWS Trainium supports both training and inference, despite its training-oriented name, and AWS reports 30–40% better price performance on average than other hardware vendors within AWS. Anthropic is training models at Amazon's largest AI data center using half a million Trainium2 chips.
• Edge AI chips run models locally on devices such as smartphones and laptops instead of sending every task to a cloud data center. Dedicated accelerators such as neural processing units can reduce communication delays and preserve privacy by keeping personal data on the user's device.

Questions & Answers

Q: How do Nvidia GPUs differ from custom AI ASICs?

Nvidia GPUs function like flexible parallel-computing workhorses that can execute many types of AI workloads, including both training and inference. Custom ASICs are hard-wired for specific mathematical operations, making them faster and more power-efficient for those defined tasks. The tradeoff is flexibility: a GPU can be reprogrammed for changing workloads, while an ASIC cannot be changed after it has been carved into silicon.

Q: Why are GPUs effective for artificial intelligence?

GPUs contain thousands of smaller cores that perform many mathematical operations simultaneously. They were originally designed for parallel graphics calculations, such as processing numerous pixels at once, but the same capability works well for neural networks. GPUs efficiently handle matrix multiplication and tensors, allowing them to accelerate both training, where models learn patterns from data, and inference, where trained models process new information.

Q: What roles do CPUs play in GPU-based AI systems?

AI workloads accelerated by GPUs or custom ASICs still often require a host central processing unit. CPUs have a relatively small number of powerful cores designed to run sequential, general-purpose tasks, while GPUs have thousands of smaller cores focused on parallel mathematics. Nvidia's Grace-Blackwell rack system illustrates this relationship by combining host CPU capabilities with interconnected GPUs optimized for advanced AI computation.

Q: Why are hyperscalers developing custom AI chips?

Major cloud providers develop custom ASICs to lower the cost and power requirements of AI, gain more control over the workloads they build, and reduce reliance on Nvidia. Their massive computing scale can justify development costs that reach at least tens and often hundreds of millions of dollars. Even so, hyperscalers continue working with Nvidia and AMD because they still need additional capacity and flexible hardware.

Q: How do Google TPUs compare with Amazon Trainium chips?

Google's TPU is described as a large factory conveyor belt with a rigid grid specialized for matrix mathematics. Amazon's Trainium resembles a cluster of workshops containing many smaller, flexible tensor engines. Although its name emphasizes training, Trainium can serve both training and inference. AWS reports that Trainium provides 30–40% better price performance on average than other hardware vendors available through AWS.

Q: What makes Nvidia's AI platform difficult to replace?

Nvidia combines GPUs with CUDA, its proprietary software platform, and sells complete rack-scale systems rather than focusing only on individual chips. Its 72-GPU Blackwell rack allows the processors to act as one GPU and is designed to improve system-level speed, power use, and performance. Cloud providers also rent Nvidia capacity to customers, reinforcing its role across hardware, software, systems, and access.

Q: How does Broadcom participate in the custom AI chip market?

Broadcom acts as a backend design partner for hyperscalers that do not want to build complete internal silicon teams. It supplies intellectual property, chip-design knowledge, and networking expertise. Broadcom helped build Google's TPUs and Meta's training and inference accelerator, and it reached a deal to help OpenAI develop custom ASICs starting in 2026. The transcript estimates its market share at 70–80%.

Q: Why are on-device AI chips useful?

On-device AI chips allow models to run locally on smartphones, laptops, and other personal hardware rather than communicating with a cloud data center for every operation. A dedicated neural processing unit can be integrated into the device's primary processor to provide AI acceleration. Local execution can preserve the privacy of information stored on a phone while reducing dependence on remote data-center communication.

Summary & Key Takeaways

• GPUs became central to AI because their thousands of smaller cores can perform parallel calculations such as matrix multiplication. Unlike CPUs, which use fewer powerful cores for sequential general-purpose tasks, GPUs efficiently support both model training and inference. Nvidia sells individual chips, complete rack-scale systems, and rented computing capacity.

• Custom ASICs sacrifice flexibility for speed, efficiency, and lower operating costs on predetermined workloads. Google pioneered cloud AI ASICs with its TPU, while AWS developed Inferentia and Trainium. Hyperscalers pursue these chips to control their infrastructure and reduce Nvidia dependence, but they continue buying GPUs to satisfy capacity and workload requirements.

• The AI chip market also includes design partners, specialized startups, and processors for edge devices. Broadcom and Marvell help hyperscalers develop custom silicon, while companies such as Cerebras and Groq pursue distinctive architectures. On-device accelerators can run models locally, reducing communication with data centers and helping preserve personal data privacy.