How to decide between fine tuning and RAG for LLMs

TL;DR

Fine tuning is not always the best path as general models improve and alternative customization methods emerge. Use a base model first, apply prompt and context engineering, add RAG for fresh or proprietary knowledge, and consider agent skills before opting for fine tuning, which is most useful for latency sensitive or highly specific tasks.

Transcript

Is fine-tuning large language models still needed today? Well let's take a well-known legal AI company because back in 2023 they fine- tuned their own model, their own custom AI, in partnership with the Frontier Lab, and in blind tests attorneys preferred this fine-tune model over the Frontier model at the time, which was GPT-4. They preferred it 9... Read More

Key Insights

• A base model is trained on large, general data and is the starting point for customization.
• Fine tuning customizes weights by training on focused data to specialize behavior, but it may be outperformed by newer general models.
• Context windows have grown large, enabling models to handle more context without weights changes.
• Reasoning and inference time are evolving, reducing the advantage of weight based fine tuning.
• RAG retrieves relevant documents at query time and feeds them into prompts, avoiding weight changes.
• Prompt and context engineering package system prompts, data, and guidelines for better guidance.
• Agent skills bundle procedural knowledge and tools that can be loaded on demand for task execution.
• LoRa and related parameter efficient methods enable lightweight fine tuning with minimal weight changes.

Questions & Answers

Q: What is the main takeaway about fine tuning versus modern customization methods?

The main takeaway is that fine tuning is not always necessary because large language models have become smarter and bigger context windows, which reduces the value of embedding knowledge into weights. Instead, methods like retrieval augmented generation, context engineering, and agent skills can tailor a model to a specific task without changing weights, offering flexibility and lower cost.

Q: When might fine tuning still be useful according to the video?

Fine tuning remains useful when latency is critical, such as real time applications like voice agents, where a small, tuned model can respond faster. It can also be beneficial for distillation, where a large teacher model guides training of a smaller model, or in reinforcement fine tuning when outputs can be programmatically graded and improved through iterative feedback.

Q: What is RAG and how does it compare to fine tuning?

RAG stands for retrieval augmented generation. It retrieves relevant documents at query time and feeds them into the prompt instead of modifying the model weights. This allows a general model to access up to date or domain specific information without training on that data, making it a flexible alternative to fine tuning in many scenarios.

Q: What role do context prompts play in customizing LLMs?

Context prompts, or context engineering, bundle system prompts, data, and formatting guidelines into a carefully crafted prompt. This approach guides the model to behave in a desired way without altering its underlying weights, enabling more precise and task aligned outputs while keeping the model general.

Q: How do agent skills contribute to customization without fine tuning?

Agent skills are folders of files that package procedural knowledge and tools. The model loads these skills on demand when it encounters a task, effectively giving the model capability to perform specialized operations without any changes to the base model weights.

Q: What is LoRa and how does it relate to fine tuning?

LoRa, or low rank adaptation, is a method that allows fine tuning by training a small adapter on top of a base model. Most fine tuning in production today is actually a form of LoRa or other parameter efficient methods, which keeps the original weights largely intact while adding new task focused capabilities.

Q: What factors should be considered in a practical decision framework for fine tuning?

Consider starting with a base model and applying prompt and context engineering. If knowledge is fresh or proprietary, add RAG, and if procedural know how is missing, add agent skills. Reserve fine tuning for when there is a specific bottleneck that other techniques cannot solve, and weigh costs of training, maintenance, and model upgrades.

Q: What is the described lineage of model customization from base to specialized?

The framework begins with a base model, then uses prompt and context engineering, adds RAG for knowledge integration, applies agent skills for procedural capabilities, and finally considers fine tuning with adapters like LoRa for narrowly defined performance needs. This layered approach helps balance performance, cost, and maintenance across evolving frontier models.

Summary & Key Takeaways

• Fine tuning is situational and may be outpaced by general model improvements, prompting the use of RAG and context engineering first. The video outlines a layered customization stack that starts with prompts and data packaging, then adds retrieval and agents, before considering adapters like LoRa. It emphasizes cost and maintenance considerations.

• RAG, context engineering, and agent skills serve as viable alternatives to weight changes in modern AI workflows, reducing the need for heavy fine tuning. The decision framework recommends starting with a base model and lightweight customization, reserving fine tuning for narrow bottlenecks or real time constraints.

• Fine tuning remains relevant but only for specific cases, such as requiring reduced latency or when distillation or reinforcement fine tuning is advantageous. The overall message is to evaluate the tradeoffs between cost, performance, and upgrade risk before committing to model weight changes.