AI Agent Fundamentals: LLMs, Embeddings, RAG Explained

TL;DR

Before building AI agents, master five fundamentals: large language models, tokens, context windows, embeddings, and orchestration through LangChain. LLMs are transformer models trained on up to tens of trillions of tokens, but they cannot see private data like a company's 500GB of documents. Embeddings convert meaning into vectors so systems retrieve relevant content by intent, not exact keywords.

Transcript

A lot has been going on with AI over the past few years. Prompt engineering, context, windows, tokens, embeddings, rag, vector DB, MCPS, agents, lang chain, langraph, claude, Gemini, and more. If you felt left out, this is the only video you'll need to watch to catch up. In this video, we assume you know absolutely nothing and try to explain all of... Read More

Key Insights

• Large language models like OpenAI's GPT, Anthropic's Claude, and Google's Gemini are all transformer models trained on large data sets, with training tokens reaching up to tens of trillions across thousands of domains such as healthcare, law, coding, and science.
• A context window is the short-term memory holding conversation history, measured in tokens, where one token is roughly three-quarters of a word for English text, and its upper limit varies by model.
• Context window sizes differ widely: xAI's Grok 4 holds 256,000 tokens, Anthropic's Claude Opus 4 holds 200,000 tokens, and Google's Gemini 2.5 Pro holds 1 million tokens, so choosing the right model for a task matters.
• Smaller nano, mini, and flash models offer context windows of only 2,000 to 4,000 tokens (about 1,500 to 3,000 words) but deliver lower latency, making them best for small documents needing fast responses.
• LLMs struggle with irrelevant context inside the window, similar to reciting many memorized digits; the apple farm problem shows a model must filter facts like color and taste to correctly total 16 apples.
• Embeddings transform text into vectors, typically 1536 numbers representing meaning, so semantically similar phrases like 'employee vacation policy' and 'staff time off guidelines' end up mathematically close even with different words.
• An agent differs from an LLM because an LLM is a static brain answering from training data, while an agent has autonomy, memory, and tools to self-determine and perform whatever task completes a request.
• LangChain is an abstraction layer that builds AI agents with minimal code, providing pre-built components for chat models, memory management, vector database integration, embeddings, and tool integration through standardized interfaces.

Questions & Answers

Q: What is a context window in a large language model?

A context window is the LLM's short-term memory that stores conversation history during a session, keeping all context in memory. It is measured in tokens, where one token is roughly three-quarters of a word for English text. The window has an upper size limit that varies by model, and once conversation content exceeds it, the model cannot hold everything at once. It functions like adding data to the conversation so the model can reference documents or prior messages while answering.

Q: How many tokens do different AI models support in their context windows?

Context window capacity varies significantly by model. xAI's Grok 4 supports 256,000 tokens, Anthropic's Claude Opus 4 supports 200,000 tokens, and Google's Gemini 2.5 Pro supports up to 1 million tokens. Smaller nano, mini, and flash models offer only 2,000 to 4,000 tokens, which equals about 1,500 to 3,000 words. Larger models like GPT-4.1 and Gemini 2.5 Pro reach 1 million tokens, roughly 7,500 words or 50,000 lines of code, so the right choice depends on the task.

Q: What are embeddings and why are they important for AI?

Embeddings transform information by converting meaning into numbers instead of storing text as words. An embedding model takes text and turns it into a vector, typically 1536 numbers that represent that meaning. Similar concepts produce similar number patterns, so 'vacation' and 'holiday' have vectors mathematically close to each other. This lets a system find relevant documents based on what someone means rather than exact words. For example, asking 'Can I wear jeans to work?' can surface the dress code policy even if it never mentions jeans.

Q: What is the difference between an LLM and an AI agent?

When you use large language models like GPT, Claude, and Gemini directly, you use them as a static brain that answers questions based on their training data. An agent, by contrast, has autonomy, memory, and tools to perform whatever task it decides is necessary to complete your request. For a customer support scenario, an agent asked about refunding a damaged product self-determines how to answer autonomously, unlike traditional software that relies on conditional statements to decide how the program executes.

Q: Why can't you just pass a company's entire document set to an LLM?

A company like TechCorp may have 500GB of internal documents that were never part of the model's training data, so the LLM has no built-in knowledge of them. Even the largest context window, Gemini 2.5 Pro's 1 million tokens, can hold only about 50 typical business files at once. The model can see just a tiny fraction of the data at any moment. This limitation is why embeddings and semantic search become necessary to retrieve only the relevant portions of a large knowledge base.

Q: What is LangChain and what problems does it solve?

LangChain is an abstraction layer that helps you build AI agents with minimal code. It addresses pain points like storing chat messages, maintaining conversation context, connecting to an internal knowledge base, and switching between providers such as OpenAI, Anthropic, or Google. It offers pre-built components including chat models for direct LLM access, memory saver for chat history, vector database integration with Pinecone or Chroma DB, OpenAI embeddings for text-to-vector conversion, and custom tool definitions for external systems, avoiding building all that infrastructure yourself.

Q: How does LangChain make it easy to switch AI providers?

LangChain provides standardized interfaces so changing providers takes one line of code. To use OpenAI you set up a chat model with model equals GPT-3 turbo, and if requirements change to Anthropic you simply switch to LLM equals ChatAnthropic with model equals Claude 3 sonnet. The same pattern applies to other capabilities like memory management through memory saver and vector database integration through consistent APIs whether you choose Pinecone or Chroma DB. This removes the need to write custom API integration code for each provider.

Q: Why do LLMs struggle with irrelevant information in a prompt?

LLMs can hold context but have practical limits on treating what is inside the window, similar to how a person struggles to recite many memorized digits at once. The video's apple farm example shows this: Sally has 14 apples, Bob has two green apples, and extra facts about apples being red or tasting bad are irrelevant to the total of 16. The model must filter distracting context to reach the answer, since color and taste have nothing to do with how many apples they own in total.

Summary & Key Takeaways

• Large language models became popular around ChatGPT's late 2022 release and are transformer models trained on up to tens of trillions of tokens. Because private data like TechCorp's 500GB of documents was never in training data, users must pass information to the LLM through the conversation history, which acts as short-term memory called the context window.

• Context windows are measured in tokens, roughly three-quarters of a word, and vary from 2,000 tokens in flash models to 1 million in Gemini 2.5 Pro. Even the largest window holds only about 50 typical business files, and LLMs struggle to isolate relevant facts, so 500GB cannot fit at once.

• Embeddings solve the scale problem by converting text into vectors of about 1536 numbers that capture meaning, enabling semantic search by intent. LangChain then ties everything together as an abstraction layer with pre-built chat models, memory savers, vector database connectors, embeddings, and tools, letting agents autonomously handle requests with minimal code.