The Future of AI Chips: Insights from Google TPU v4 and the Storage Chip Giants

The Future of AI Chips: Insights from Google TPU v4 and the Storage Chip Giants

Introduction:
AI chips play a crucial role in accelerating machine learning models and powering AI applications. In this article, we will explore the future of AI chips by examining the design of Google TPU v4 and the advancements in storage chip technology.

Google TPU v4: Optimizing Embedding Layers with SparseCore
Google has designed a dedicated acceleration module called SparseCore (SC) in TPU v4 to optimize embedding layers. Each SC unit consists of a vector processing unit (scVPU), 2.5 MB local SRAM, and a memory access interface that can access up to 128TB of shared HBM. Additionally, SC units include specialized acceleration logic for embedding layer operations such as sorting, reduction, and concatenation. Despite their simple structure, a large number of SC units are deployed in each TPU v4, accounting for only about 5% of the overall area and power consumption. Comparing the performance of running embedding layers on CPU versus TPU v4 SC, Google's research shows that placing embedding layers on TPU v4 SC can improve the overall recommendation system's speed by more than 6 times.

Domain-Specific Design and Reconfigurable Optical Interconnects
Different machine learning models have varying data flow requirements, broadly classified into data parallel, model parallel, and pipeline parallel. With reconfigurable optical interconnects, the interconnection topology between TPUs can be adjusted based on the specific model's data flow, resulting in optimal performance improvements of over 2 times. This flexibility allows for efficient utilization of resources and enhances overall system performance.

Reliability and Scalability with Reconfigurable Optical Interconnects
In a supercomputer composed of a massive number of chips, ensuring high performance even with a small portion of faulty chips is crucial. Reconfigurable optical interconnects enable bypassing the faulty chips, maintaining the system's overall performance with minimal impact. Google's research demonstrates that with a 99% chip reliability rate, using reconfigurable optical interconnects and optical cross switches can improve the average system performance by up to 6 times.

The Shift Towards Large Models and Optimization for Recommendation Systems
As AI progresses, the focus has shifted towards handling large models effectively. For Google, optimizing recommendation systems is crucial as they determine the company's primary revenue streams. The current bottleneck in recommendation systems lies in the embedding layer, responsible for mapping high-dimensional sparse features to low-dimensional dense features. Google's TPU v4 design addresses this bottleneck by optimizing the embedding layer computation, which primarily involves 1D vector computations and efficient data exchange across distributed chips.

The Common Ground: Scalability and AI Chip Design
While different chip companies may emphasize various aspects of AI chip design, such as chiplets for scalability or technologies like NvLink for multi-card scalability and performance improvement, the common goal is to support scalability for large models. In the future, scalability, including data interconnect bandwidth, may become as crucial as peak computing power in AI chip design. This trend calls for expertise in multiple domains, including digital logic, computer architecture, packaging, and data interconnects.

The AI Chip Power Race: Storage Chip Giants Enter the Fray
NVIDIA and AMD have approached SK Hynix for samples of the upcoming HBM3E chip, highlighting the increasing demand for high-end servers driven by generative AI. SK Hynix currently holds a 50% market share in the global HBM market, followed closely by Samsung with 40% and Micron, a US storage chip giant, with 10%. The demand for high-density DDR5 and HBM products has surged in response to the growing need for generative AI.

Conclusion:
As AI continues to evolve, the future of AI chips lies in optimizing performance, scalability, and reliability. The advancements in Google TPU v4, with its domain-specific design and reconfigurable optical interconnects, provide valuable insights into the direction of AI chip development. Additionally, the involvement of storage chip giants like SK Hynix, Samsung, and Micron reflects the intensified competition and the increasing importance of storage solutions in the AI chip landscape.

Actionable Advice:

1. Embrace domain-specific design: Tailor your AI chip designs to the specific requirements of the target application or workload, optimizing performance and power efficiency.
2. Invest in reconfigurable interconnects: Explore the benefits of reconfigurable optical interconnects to enhance scalability, reliability, and performance in your AI chip designs.
3. Foster cross-domain expertise: Develop a multidisciplinary approach to AI chip design, encompassing digital logic, computer architecture, packaging, and data interconnects to meet the evolving demands of large-scale AI models.

By combining the insights from Google TPU v4 and the developments in storage chip technology, the future of AI chips holds the promise of unlocking unprecedented performance, scalability, and efficiency for AI applications.