Accelerating the Breakthrough of Domestic AI Chips: A Journey of Compatibility and Optimization

Accelerating the Breakthrough of Domestic AI Chips: A Journey of Compatibility and Optimization

Introduction:
The development of AI chips has become a crucial focus for industries and countries worldwide. To achieve breakthroughs in this field, a team from the Chinese Academy of Sciences (CAS) is making significant strides by prioritizing algorithm sacrifice and chip selectivity. Their approach aims to address the challenges of compatibility and optimization, which are key factors in the success of domestic AI chips.

Compatibility: Embracing New Languages and Ecosystems:
In the pursuit of creating AI chips that can compete with established players, hardware manufacturers are presented with a short-term shortcut - compatibility with popular frameworks like CUDA. However, the CAS team suggests that in order to truly revolutionize the industry, embracing new languages and ecosystems such as Triton and SYCL is essential. These emerging languages may represent the future of AI chip development, offering unique advantages and paving the way for innovative solutions.

Optimization: Large Model Inference and Computing Efficiency:
One critical aspect of AI chip development is optimizing large model inference and computing efficiency. To illustrate this, let's take the example of the LLaMA-65B model, which consists of 64 attention heads. Suppose we have four computing cards available for processing a batch of inputs and multiplying them with the QKV weight matrix.

To achieve parallel processing, the CAS team utilizes model parallelism by dividing the QKV weight matrix vertically into equal parts and distributing them across the four computing cards. This distribution enables each computing card to independently handle 16 attention heads, eliminating the need for inter-card data communication. Once each card receives its portion of 16 heads, they are further distributed to different sub-computing units for parallel processing. After the attention calculations are complete, each computing card combines the outputs of its 16 heads. By following this approach, there is no need for inter-card data exchange.

In comparison to tensor parallelism, where batch size is divided by N, the throughput of both methods remains the same, assuming equal latency. Although tensor parallelism may offer superior total latency due to the simultaneous reading of weights by all computing cards, the bandwidth for reading weights in pipeline parallelism remains limited to the memory bandwidth of a single computing card.

Insights and Unique Ideas:
The CAS team's approach showcases the importance of compatibility with new languages and ecosystems, as well as the significance of optimizing large model inference and computing efficiency. By embracing emerging languages, hardware manufacturers can stay ahead of the curve and unlock the full potential of domestic AI chips. Additionally, the utilization of model parallelism and pipeline parallelism provides insights into effective strategies for handling large models and enhancing computing efficiency.

Actionable Advice:

1. Embrace emerging languages and ecosystems: Instead of solely relying on compatibility with established frameworks, explore new languages like Triton and SYCL. These languages may offer unique features and advantages that can propel the development of domestic AI chips.

2. Optimize model parallelism and pipeline parallelism: When dealing with large models, consider implementing model parallelism to distribute the weight matrices across computing cards. Additionally, explore the benefits of pipeline parallelism, which can enhance computing efficiency by allowing independent processing of subsets of the model.

3. Prioritize hardware compatibility and optimization: To achieve breakthroughs in domestic AI chips, hardware manufacturers should prioritize both compatibility with existing frameworks and the optimization of computing efficiency. By striking a balance between the two, manufacturers can create chips that are both versatile and high-performing.

Conclusion:
The journey towards the breakthrough of domestic AI chips requires a focus on both compatibility and optimization. By embracing new languages and ecosystems, hardware manufacturers can stay at the forefront of innovation. Additionally, optimizing large model inference and computing efficiency through model parallelism and pipeline parallelism can significantly enhance chip performance. By incorporating these strategies, the CAS team and other researchers are accelerating the path towards achieving groundbreaking advancements in domestic AI chip development.