How to Distill Large Models Into Smaller Models

TL;DR

Distillation transfers task-specific behavior from a large model to a smaller, cheaper model by fine-tuning it on the larger model’s outputs. The process requires custom evaluations, stored input-output examples, and fine-tuning, allowing developers to test whether a small model can deliver comparable quality for a narrow production task while improving cost efficiency and scalability.

Transcript

Hello, everyone, and welcome to Dev Day London. As you know, the past year has been a very exciting one for generative AI. In twenty twenty-three, we released GPT-4. It was the first time that most people had a chance to directly interact with a powerful artificial intelligence. And developers like you rapidly embraced AI, building prototypes and e... Read More

Key Insights

• Production AI requires sustainable operation because a successful application must remain available, avoid disruptive rate-limit constraints, respond with acceptable latency, and maintain viable costs. These concerns become more important when an application gains users, makes several model calls in sequence, or supports real-time interactions.
• Model selection balances pricing and intelligence because smaller models can cost substantially less while larger models provide broader capabilities. The transcript compares GPT-4.0 at around four dollars per one million tokens with GPT-4.0 Mini at around twenty-four cents, making the smaller model more than fifteen times cheaper.
• Academic benchmarks measure broad model capabilities, but they do not necessarily reflect how users experience a particular application. For a review-classification system, the meaningful measure is the percentage of reviews assigned to the correct category, rather than performance on broad benchmarks such as MMLU or GPQA.
• Distillation transfers task-specific intelligence by fine-tuning a smaller model on outputs produced by a larger model. This process resembles compression because it places the capabilities needed for a narrow domain into a smaller model without attempting to reproduce all of the larger model’s broad intelligence.
• Task-specific evaluations are the first requirement for distillation because improvement cannot be established without measurement. An evaluation should directly represent the application’s desired behavior, such as the percentage of reviews classified correctly, and should support comparisons among the large model, the original small model, and fine-tuned versions.
• Training examples come from recording how a large model responds to real-world application prompts. The resulting dataset contains input-output pairs in which the inputs reflect actual use cases and the outputs demonstrate the behavior that the smaller model should learn during fine-tuning.
• Fine-tuning teaches the smaller model to imitate the larger model on the selected task. The resulting task-specific model can then be evaluated for possible production deployment, where it may provide a good user experience at a lower cost than repeatedly using the larger model.
• OpenAI’s distillation workflow combines stored completions, custom evals, and the fine-tuning API. Setting the Chat Completions API store parameter to true preserves inputs and outputs, while the evals product supports custom tests and model comparisons through the platform.

Questions & Answers

Q: What is model distillation for AI applications?

Model distillation is a technique for transferring task-specific behavior from a large model to a smaller model. Developers collect outputs generated by the large model for representative application inputs, then fine-tune the smaller model on those input-output pairs. The goal is to approach the larger model’s quality within a narrow domain while gaining the smaller model’s lower production cost.

Q: How does model distillation work step by step?

Model distillation follows three principal steps. First, developers create evaluations that measure performance on the application’s actual task. Second, they capture strong completions from a large model for representative real-world prompts, producing a dataset of input-output pairs. Third, they fine-tune a smaller model on that dataset and use the evaluations to determine whether its performance is suitable for deployment.

Q: Why should developers create evaluations before fine-tuning?

Developers should create evaluations first because they need a measurable definition of success before changing the model. The evaluation should represent the production task directly, such as the percentage of reviews assigned to the correct category. It then provides a consistent basis for comparing the large model, the unmodified small model, and any fine-tuned models trained through distillation.

Q: Why are academic AI benchmarks insufficient for production decisions?

Academic benchmarks such as MMLU, GPQA, and GSM8K measure broad intelligence and help compare models overall, but they may not correspond to an application’s real task. A production system might classify reviews, extract financial information, or generate email replies. For these narrow uses, developers need custom evaluations that measure the behavior users actually experience rather than relying only on a general benchmark score.

Q: How can distillation reduce the cost of an AI application?

Distillation can reduce cost by moving a narrow production task from a large model to a fine-tuned smaller model. The transcript compares GPT-4.0 at around four dollars per one million tokens with GPT-4.0 Mini at around twenty-four cents, a difference of more than fifteen times. Savings are useful only if evaluations confirm that the smaller model still meets the application’s quality requirements.

Q: What data is needed to distill a large model into a small model?

Distillation requires representative input-output pairs. The inputs should be real-world prompts from the application, while the outputs should be strong responses generated by the large model. These examples show the smaller model what good task performance looks like. Dataset creation is especially important because the quality and relevance of the examples determine what behavior the smaller model is taught during fine-tuning.

Q: How do stored completions support model distillation?

Stored completions simplify collection of training examples through the Chat Completions API. By setting the store parameter to true, developers can persist the inputs and outputs from completion calls. They can then inspect those records in the dashboard and create datasets from them, avoiding the need to design a separate database and application-writing process solely for collecting distillation examples.

Q: When is model distillation appropriate for a production application?

Model distillation is appropriate when an application needs strong performance on a specific, narrow task rather than the full breadth of a large model’s intelligence. Examples in the transcript include review classification, financial information extraction, and email reply generation. Developers should consider deployment only after task-specific evaluations show that the fine-tuned smaller model provides an acceptable experience for users.

Summary & Key Takeaways

• Production AI applications must operate reliably, affordably, and at scale, so model selection involves more than proving that a concept works. Developers must consider uptime, rate limits, latency, and cost. A model’s unit economics are particularly important because spending more on model calls than an application earns is not sustainable.

• Large models offer broad intelligence across many domains, but production applications often need strong performance on narrow tasks such as classifying reviews, extracting financial information, or generating email replies. Distillation addresses this mismatch by using outputs from a capable large model to train a smaller model for a specific application domain.

• The distillation workflow has three core stages: create task-specific evaluations, capture strong completions from a large model, and fine-tune a smaller model on those input-output pairs. OpenAI’s stored completions, evals product, and fine-tuning API support this workflow by simplifying dataset creation, model comparison, training, and performance tracking.