Which LLM Is Best for RAG: GPT-4o, Claude or Gemini?

13.4K views
•
January 30, 2025
by
Nate Herk | AI Automation
YouTube video player
Which LLM Is Best for RAG: GPT-4o, Claude or Gemini?

TL;DR

Gemini Flash 2.0 was the fastest model for RAG in an n8n test, answering in 6.7 seconds versus 11 seconds for GPT-4o and almost 21 seconds for Claude 3.5 Sonnet. On information recall, Claude 3.5 Sonnet scored highest with 9 out of 10, ahead of GPT-4o at 7 and Gemini at 6. The grader was ChatGPT, given the source PDF.

Transcript

wow flash was 6.7 seconds GPT was 11 seconds and then open AI or anthropic CLA was almost 21 seconds so that's that's kind of a big jump here so we all know that rag is so so important when we're building out our AI agents but the question is which large language model is actually the best for rag So today we're going to be answering this question ... Read More

Key Insights

  • RAG stands for retrieval augmented generation: an agent retrieves information that is not in its training data or prompt, augments it with the context of the query, and then has the large language model generate a human-readable response.
  • In a RAG workflow, the language model turns the user's question into a query for the vector database. The query is vectorized, the nearest vectors are retrieved, and that content goes back to the model to produce the answer.
  • Keeping a model comparison fair means giving every agent the same system prompt and the same vector database with the same information, and sending each one the identical test prompt, so the model is the only variable.
  • Using ChatGPT with the source PDF as a blind grader gives consistent scores: responses are submitted without saying which model wrote them, and the order is shuffled between tests to reduce bias.
  • Claude 3.5 Sonnet scored 9 out of 10 on information recall because its answer was more detailed and better structured, while GPT-4o scored 7 and Gemini Flash 2.0 scored 6, even though all three were factually correct.
  • Gemini Flash 2.0 was the fastest model in the speed test at 6.7 seconds, compared with 11 seconds for GPT-4o and almost 21 seconds for Claude 3.5 Sonnet, a large gap for latency-sensitive agents.
  • A good RAG agent can split a vague, multi-part question into separate queries: for one question, Claude sent two queries to the vector store, one about segment growth and one about dividend policy changes.
  • Reading an n8n agent's execution log shows how the model reasoned, which query it sent to the vector store, what came back, and how it shaped the final answer, which helps explain differences between models.

Explore YouTube Video Summarizer or Get YouTube Transcript Extractor

Questions & Answers

Q: Which LLM is fastest for RAG: GPT-4o, Claude 3.5 Sonnet or Gemini Flash 2.0?

In an n8n RAG experiment with all three models querying the same Nvidia financial data in a vector database, Google's Gemini Flash 2.0 was the fastest, responding in 6.7 seconds. OpenAI's GPT-4o took 11 seconds, and Anthropic's Claude 3.5 Sonnet took almost 21 seconds. That is a large difference to weigh when choosing a model for responsive AI agents.

Q: What is retrieval augmented generation (RAG)?

Retrieval augmented generation is the process of an AI agent fetching information it does not already have in its training data or prompt. Retrieval means fetching data from a vector database or other store. Augmented means combining that data with the user's query and the prompt. Generation means the large language model writes a human-readable response based on its role and the original query.

Q: How does a RAG agent query a vector database in n8n?

The user submits a query, and the agent reads its system prompt and decides to call its vector store tool. The model turns the question into a query, which is vectorized with an embeddings model. The nearest vectors are pulled back as retrieved content, and the model then structures that content into a readable answer. The agent's log in n8n shows each of these steps.

Q: How were GPT-4o, Claude 3.5 Sonnet and Gemini Flash 2.0 tested for RAG?

Each model ran as a separate n8n agent with the same system prompt: retrieve and summarize Nvidia's financial and earnings information. All three agents used the same Pinecone vector database. Each test sent the identical prompt to all three. The responses were then given, unlabeled, to ChatGPT, which had the original PDF and rated each one out of 10.

Q: What criteria matter when comparing LLMs for RAG?

The experiment used seven criteria: information recall, query understanding, response coherence and completeness, speed, context window management, handling conflicting information, and source attribution. Together they cover whether a model retrieves the right facts, understands vague or multi-part questions, writes complete answers, responds quickly, handles large contexts, resolves contradictions, and cites where its information came from.

Q: Which model scored best on information recall in the RAG test?

Claude 3.5 Sonnet scored highest on information recall with 9 out of 10. The test asked how much Nvidia's GAAP operating income grew year over year in Q1 fiscal 2025. All three models answered correctly with 690%, but Claude's answer was more detailed and better structured. GPT-4o scored 7 and Gemini Flash 2.0 scored 6, even though GPT-4o and Gemini gave identical answers.

Q: Why use ChatGPT as a grader when comparing RAG model outputs?

Scoring each response out of 10 by hand was hard and subjective, so ChatGPT, with access to the original source PDF, was used as one consistent grader. It was not told which model wrote which response, and the order of the responses was changed between tests. The creator acknowledged this is not a perfect scientific method, but it reduced personal interpretation.

Q: How do LLMs handle vague, multi-part questions in RAG?

For the query understanding test, the agents got a vague, two-part question: tell me about growth in Nvidia's segments and how Nvidia's dividend policy is changing. The execution log showed that Claude split it into two separate queries to the vector store, one on segment growth and one on dividend policy. The response structure and depth differed noticeably between the three models.

Summary & Key Takeaways

  • Three n8n AI agents, using GPT-4o, Claude 3.5 Sonnet and Gemini Flash 2.0, were given the same system prompt and the same Pinecone vector database of Nvidia financial and earnings information. Each was tested on seven RAG criteria to see which large language model performs best for retrieval augmented generation.

  • The seven criteria were information recall, query understanding, response coherence and completeness, speed, context window management, conflicting information, and source attribution. Each agent received the exact same prompt per test. ChatGPT, with access to the original PDF, graded the unlabeled responses out of 10 to keep scoring consistent.

  • On information recall, all three models correctly reported that Nvidia's GAAP operating income grew 690% year over year in Q1 fiscal 2025. Claude scored 9 for its detail and structure, GPT-4o scored 7 and Gemini scored 6. On speed, Gemini Flash took 6.7 seconds, GPT-4o took 11 seconds and Claude took almost 21 seconds.


Read in Other Languages (beta)

Share This Summary 📚

Explore More Summaries from Nate Herk | AI Automation 📚