Accelerating Generative AI with PyTorch II: GPT, Fast - AI芯片，看什么？

Accelerating Generative AI with PyTorch II: GPT, Fast - AI芯片，看什么？

Introduction:
In the world of machine learning, the performance of AI chips is a fundamental limitation. One crucial factor to consider is power consumption. While the H100 chip theoretically has the capability to achieve 2,000 TFLOPS of computing power, it faces power limitations before reaching that point. Therefore, tracking the FLOPs per joule of energy is essential. With modern training runs often exceeding 1e25 flops, highly efficient chips are needed to absorb megawatts of power within months to outperform the state of the art (SOTA) models.

Two Objectives in ML Quantization Research:
Within the vast design space, most of the current research on machine learning quantization boils down to two objectives:

1. Efficiently storing billions of weights while using as few bits as possible, reducing memory consumption from the perspectives of capacity and bandwidth. This objective depends on the digital format used to store the weights.

2. Achieving good energy and area efficiency. This primarily depends on the digital format used for weights and activations. While sign-magnitude representation is intuitive, it is not efficient as it requires implementing completely different addition and subtraction algorithms, which are distinct from circuits for unsigned integers without a sign bit. Interestingly, hardware designers can solve this problem by using two's complement representation, which allows the use of the same carry adder circuit for positive, negative, and unsigned numbers. All modern CPUs use two's complement representation. As a trick to share hardware resources for INT8 and UINT8, -1 can be represented as 11111111. When the number is incremented by 1, it overflows to 00000000, representing 0 as expected. Similarly, -2 can be represented as 11111110. Overflow is used as a feature! In fact, the range of 0 to 127 is mapped to normal values, while the range of 128 to 255 is directly mapped to -128 to -1. FP8 (1,5,2 or 1,4,3) recently standardized some additional quirks in the OCP standard, but there is currently no consensus. Many AI hardware companies have implemented chips with slightly superior variants that are incompatible with the standard.

The Debate on FP8 FMA vs. INT8 FMA:
In summary, this article, along with others, claims that FP8 FMA occupies 40-50% more silicon area than INT8 FMA and has equally higher or worse energy consumption. This is the primary reason why most dedicated ML inference chips use INT8. NF4 and its variants (AF4) are 4-bit formats that assume weights follow a completely normal distribution and utilize precise lookup tables to minimize errors. However, this approach is very expensive in terms of area and power consumption - every operation now requires a lookup table with a huge number of entries, which is much worse than any INT/FP operation. Hardware vendors face a tricky problem of trying to design highly specialized and efficient formats without closing the door to future model architectures that may have completely different value distributions.

The Sensitivity of Inference to Cost/Power Consumption:
Inference is particularly sensitive to cost/power consumption because models are typically trained only once but deployed to millions of customers. Training is also more complex, involving many operations with numerical issues (see below). This means that inference chips usually lead the way in adopting smaller, cheaper digital formats, which can result in a significant difference between the format used for model training and the format used for model inference.

Actionable Advice:

1. Use LLM.int8() to convert most exceptional value weights to INT8 while preserving a small portion of them. This helps achieve better quantization.

2. Utilize GPTQ to incorporate second-order information about weight matrices for improved quantization. This technique leverages additional insights to achieve better results.

3. Implement Smoothquant to mathematically transform activations and attempt to smooth out exceptional values. This helps in achieving more accurate quantization of the most significant weights.

Conclusion:
In conclusion, AI chips are essential for accelerating generative AI models like GPT. Power consumption is a significant limitation, and finding efficient ways to store weights and perform computations is crucial. While there is ongoing debate between FP8 FMA and INT8 FMA, most dedicated ML inference chips currently use INT8 due to its efficiency. Hardware vendors face the challenge of designing specialized formats without hindering future model architectures. It is crucial to consider cost and power consumption in inference, as models are deployed to millions of customers. By following actionable advice such as LLM.int8(), GPTQ, and Smoothquant, developers can optimize their models for efficient and accurate quantization.

(Note: The content has been combined and rephrased to create a coherent long-form article. Unique insights and ideas have been incorporated, and three actionable advice points have been added for a more informative conclusion.)