"Optimizing Inference Performance of Language Models"

"Optimizing Inference Performance of Language Models"

Introduction:
In the field of natural language processing, language models have become increasingly complex and powerful. However, with the increase in model size, the inference performance of these models becomes a critical consideration. In this article, we will explore the factors that affect the inference performance of language models and discuss strategies to optimize their efficiency.

Determining Input and Output Sizes:
To establish a "typical" input and output size, we can analyze the user data from Anyscale Endpoints. Based on this data, we have selected the following values:

• Average input length: 550 tokens (with a standard deviation of 150 tokens)
• Average output length: 150 tokens (with a standard deviation of 20 tokens)

When calculating the token count, we consistently use the Llama 2 tokenizer, which provides an independent estimation of token count. It is worth noting that previous research has shown that the tokenizer used in ChatGPT is more "efficient" compared to Llama 2 (with ChatGPT having 1.33 tokens per word and Llama 2 having 1.5 tokens per word). Therefore, we believe that ChatGPT should not be penalized for this difference.

Factors Affecting Forward Pass of Language Models:
In the past, dynamic batching (waiting for multiple requests to synchronize processing) was used to improve GPU utilization. However, this approach has its drawbacks as it often requires padding inputs to the same length or waiting for a larger batch to be constructed. To efficiently schedule the forward pass, we need to understand the independent variables to consider during the scheduling process.

We have observed that the composition of sequences (batch size within the sequence) in the forward pass has a negligible impact on performance. This means that we can build an efficient scheduler around a single variable, namely the number of tokens in the forward pass.

Relationship Between Throughput and Token Count:
A language model has two crucial operating ranges, with a steep transition between them. When the token count is low, the GPU bottleneck is reading the model from memory, resulting in an increase in throughput as the token count increases. However, when the token count is high, the throughput of the model is limited by GPU computational power, resulting in a near-constant throughput. Therefore, to achieve optimal model efficiency, it is best to keep all forward passes within the saturation range of throughput.

Scheduling a Group of Tokens Across Multiple Forward Passes:
By observing the token throughput curve for well-aligned inputs, we notice that it is concave, indicating that the second derivative is less than or equal to 0. Let 𝑓(𝑥) be the concave function representing the delay to throughput ratio for a given model.

To schedule a group of tokens across multiple forward passes, the most effective approach is to evenly distribute them between two batches. In general, in a system where P tokens need to be processed across F forward passes, the ideal partitioning scheme is to distribute them evenly.

Actionable Advice:

1. Optimize the input and output sizes: Based on the observed user data, adjust the average input and output lengths to maximize the efficiency of the language model's inference performance.
2. Implement an efficient scheduler: Develop a scheduler that takes into account the number of tokens in the forward pass to ensure optimal performance and throughput.
3. Consider the distribution of tokens: When scheduling a group of tokens across multiple forward passes, evenly distribute them between batches to achieve the highest throughput.

Conclusion:
Inference performance is a critical aspect of language models, and optimizing their efficiency is essential. By understanding the factors that affect inference performance, such as input and output sizes and scheduling strategies, we can improve the overall performance of language models. By implementing the actionable advice mentioned above, researchers and developers can enhance the inference performance of language models and unlock their full potential in various applications.