The Future Trends in Computing Chips: Insights from the Rise of Large Models like ChatGPT and Others

The Future Trends in Computing Chips: Insights from the Rise of Large Models like ChatGPT and Others

Introduction:
In recent years, there has been a significant rise in the popularity of large models like ChatGPT, which has sparked discussions about the future trends in computing chips. These models require extensive computational power and memory capabilities to handle their complex tasks effectively. In this article, we will explore the development trends of computing chips by examining the common points between the rise of large models and the advancements in chip technology. Additionally, we will provide actionable advice for individuals and organizations to leverage these trends effectively.

The Growing Importance of High-Performance Networks:
One fundamental requirement for large models like ChatGPT is the ability to efficiently handle the massive amounts of data generated during training and inference. This data flow heavily relies on high-performance networks, which play a crucial role in enabling seamless communication between nodes in a cluster. Statistics indicate that east-west network traffic, which refers to the traffic between nodes within a data center, accounts for more than 85% of the total data flow. Moreover, in AI model training clusters with over 1000 nodes, the east-west traffic is estimated to surpass 90%.

To optimize the performance of high-performance networks, various techniques can be employed. These include congestion control, multipath load balancing using Equal-Cost Multipath (ECMP), out-of-order delivery, scalability improvements, fast fault recovery, and Incast optimization. By fine-tuning these network functionalities, it becomes possible to achieve superior high-performance network capabilities. One notable development in this domain is the integration of ultra-high bandwidth networks in chips like Gaudi. This integration significantly enhances the efficiency of east-west traffic interaction between cluster nodes, making larger-scale cluster designs feasible.

The Cost Challenges in Memory Chip Production:
While the rise of large models has demanded more computational power, it has also highlighted the challenges associated with memory chip production. For instance, the H100 PCIe and SXM versions of computing chips utilize five HBM stacks, with the H100S SXM version even featuring six stacks. However, it is the H100 NVL version, with its twelve HBM stacks, that stands out as a remarkable achievement. Nonetheless, the cost implications are significant.

According to research, the cost of a single 16GB HBM stack amounts to a staggering $240. Therefore, the cost of memory chips alone in the H100 NVL version reaches nearly $3000. Analysts have further estimated that the production cost of each H100 chip, utilizing TSMC's 4N process (5nm), involves a 12-inch wafer priced at $13,400, theoretically allowing the creation of 86 H100 chips. Considering the yield rate, TSMC can generate approximately $155 in revenue per H100 chip.

However, the actual revenue generated by each H100 chip is likely to exceed $1000 for TSMC. This is primarily due to the adoption of TSMC's CoWoS packaging technology, which brings in additional revenue of $723 per chip. CoWoS stands for Chip on Wafer on Substrate, referring to the process of assembling bare chips on a wafer and then packaging them on a substrate. Traditional packaging only involves the on Substrate (oS) step, which is typically handled by third-party packaging and testing facilities after the wafer manufacturing is completed. However, the advanced CoWoS packaging step adds complexity that cannot be handled solely by these facilities.

Despite the undeniable benefits of CoWoS, its high price range of $4000-6000 per chip has deterred many potential customers, including even affluent companies like Apple. As a result, TSMC's capacity for CoWoS production remains limited.

Insights and Actionable Advice:

1. Embrace the potential of high-performance networks: As the reliance on large models and data-intensive tasks increases, organizations should invest in optimizing their network infrastructure. By implementing techniques such as congestion control, load balancing, and fast fault recovery, they can achieve higher efficiency in east-west network traffic.

2. Consider the cost implications of memory chip production: For those involved in the development or procurement of computing chips, it is essential to carefully evaluate the costs involved in memory chip production. The adoption of advanced packaging technologies like CoWoS may bring significant benefits but must be balanced with the higher price range.

3. Collaborate with chip manufacturers: To stay ahead of the curve, it is crucial for organizations to establish collaborative relationships with chip manufacturers. By actively engaging in discussions and sharing insights on their needs and requirements, they can influence the future development of computing chips and ensure that their specific demands are addressed.

Conclusion:
The rise of large models like ChatGPT has shed light on the future trends in computing chips. The importance of high-performance networks and the challenges of memory chip production have become prominent areas of focus. By optimizing network capabilities, carefully considering cost implications, and collaborating with chip manufacturers, individuals and organizations can navigate these trends effectively. As the demand for computational power continues to grow, staying informed and proactive will be key to harnessing the full potential of computing chips in the future.