How Does RAG Retrieve and Answer Questions?

TL;DR

RAG improves an LLM’s answers by retrieving relevant information at query time, adding it to the prompt, and generating a response from that context. It is best suited to dynamic factual information, while prompt engineering controls behavior and fine-tuning shapes stable patterns such as voice and style. A complete system also requires search, embeddings, vector storage, chunking, caching, monitoring, error handling, and production deployment.

Transcript

Everyone's talking about RAG. If you feel left out, this is the only video you need to watch to catch up. In this video, we'll learn Rag in a super simplified manner with visualizations that will make it easy for anyone to understand. No background knowledge in AI or AI models or coding or programming required. We'll start with the simplest explana... Read More

Key Insights

• RAG is a process that retrieves relevant information, augments a user prompt with that information, and uses an LLM to generate a contextual response. The retrieval occurs automatically in a RAG system, so users do not need to locate and paste source passages themselves.
• Missing context is a major reason an LLM may provide an incorrect or generic answer about private information. Supplying the relevant section of an internal policy document gives the model the context required to answer a company-specific question more accurately.
• Prompt engineering is appropriate for defining chatbot rules, restrictions, and security behavior. Instructions can prohibit disclosure of personal or confidential information, limit which topics the chatbot addresses, and require sensitive questions about matters such as performance review appeals or salary discussions to be redirected to HR.
• Fine-tuning is appropriate for stable, repeating patterns such as voice, tone, style, and language. A model can be trained with sample questions and answers drawn from speeches, emails, blog posts, and videos to produce responses resembling a particular communicator.
• Fine-tuning is poorly suited to frequently changing factual information because policy updates would require retraining. Training is described as expensive and slow, trained answers do not provide source citations, larger training datasets can reduce accuracy, and the resulting knowledge is limited to the training data.
• RAG is suited to dynamic factual information because it retrieves relevant material at query time rather than embedding the information during training. This design allows a chatbot to consult changing policy documents when answering each new employee question.
• Keyword search finds and ranks documents through exact terms such as “reimbursement” and “home office.” It can identify documents containing those words, but exact matching may miss relevant content expressed with different language, which lowers the chance of consistently retrieving accurate information.
• A production-oriented RAG architecture extends beyond retrieval and generation. The covered pipeline includes semantic search, embedding models, vectors, dot products, vector databases, ChromaDB, document chunking, caching, monitoring, error handling, and deployment, with browser-based labs supporting practice after the lectures.

Questions & Answers

Q: What is retrieval augmented generation?

Retrieval augmented generation, or RAG, is a method for producing answers from information retrieved when a question is asked. A RAG system locates relevant material, adds that material to the user’s prompt, and has an LLM generate a response from the augmented context. Its three named stages are retrieval, augmentation, and generation, and the lookup is handled automatically for the user.

Q: How does RAG help an LLM answer questions about private documents?

RAG gives the LLM context that was not available in its existing knowledge. For a question about a private home office reimbursement policy, the system searches internal policy documents, retrieves the relevant section, and includes it in the prompt. The LLM can then base its response on that company-specific passage instead of producing a generic or incorrect answer without access to the policy.

Q: When should RAG be used instead of prompt engineering?

RAG should be used when an answer depends on relevant factual information that must be retrieved from documents at query time. Prompt engineering serves a different purpose: it defines how the chatbot should behave, what it may reveal, and which requests it must refuse or redirect. For example, prompt instructions can send sensitive salary or performance-review questions to HR, while RAG retrieves approved policy content.

Q: When should fine-tuning be used instead of RAG?

Fine-tuning should be used for stable patterns that the model should reproduce consistently, such as voice, tone, language, and communication style. Training examples can come from speeches, emails, blog posts, videos, and sample question-answer pairs. RAG is the better choice for changing factual content because it retrieves information when a query arrives instead of requiring the facts to be incorporated through model training.

Q: Why is fine-tuning unsuitable for frequently changing policies?

Fine-tuning is unsuitable for frequently changing policies because every meaningful update can require the model to be retrained. The transcript describes training as expensive, slow, and dependent on computational resources. It also notes that users cannot verify answers through citations, larger training data can lower accuracy, and the model only knows information included in its training data, creating a knowledge cutoff.

Q: How does keyword search retrieve documents for a RAG system?

Keyword search identifies important terms in the user’s question and looks for exact occurrences in available documents. For a home office reimbursement question, the selected terms might be “reimbursement” and “home office.” The search process counts or scores those terms across documents and ranks the documents accordingly. TF-IDF and BM25 are presented as two popular techniques for performing this type of retrieval.

Q: What components are covered in a complete RAG pipeline?

The covered pipeline begins with keyword search using TF-IDF and BM25, then moves to semantic search, embedding models, embeddings, vectors, and the dot product. It also includes vector databases, ChromaDB, and strategies for dividing documents into chunks. These pieces are brought together in a complete RAG architecture, followed by caching, monitoring, error handling, and a brief production deployment setup.

Q: Why is document chunking important in the presented RAG architecture?

Document chunking is presented as a required part of the path from stored documents to a complete RAG pipeline. The course covers what chunking is, different document chunking strategies, and a dedicated browser-based lab before building the full architecture. Within the presented sequence, chunking prepares document content for the retrieval components that locate relevant information and supply it to the model’s augmented prompt.

Summary & Key Takeaways

• Retrieval augmented generation addresses questions that require information unavailable to the underlying language model, such as private company policies. The system finds relevant passages, adds them to the user’s prompt, and asks the model to produce a contextual answer. This reduces reliance on generic responses and makes internal knowledge directly usable.

• RAG is one of several methods for improving AI responses. Prompt engineering supplies rules, restrictions, and instructions, while fine-tuning teaches stable patterns such as a particular voice or communication style. RAG is better for changing factual knowledge because it retrieves current source material when each user question is submitted to the system.

• Building a RAG pipeline involves progressively stronger retrieval techniques and supporting components. The course covers keyword search with TF-IDF and BM25, semantic search, embedding models, vectors, dot products, vector databases, ChromaDB, and document chunking. It then combines those elements into an architecture with caching, monitoring, error handling, and production deployment.