How to build a document context layer for AI agents

TL;DR

The talk argues that in 2026 RAG decomposes into an agent harness and a context layer, with context moving up the stack and retrieval baked into the agent. It covers document parsing, semantic storage, and repeatable workflows as the core for enabling agents to read and act on large document collections.

Transcript

[music] >> I think we can get started. Hey everyone, I'm Jerry, co-founder and CEO of LlamaIndex, and today I'm excited to uh give a talk called building the document context layer for AI agents. Um really uh big shoutout to the AI Engineer World Fair for hosting. Um and if you've seen some of my earlier talks from the previous uh AI Engineer confe... Read More

Key Insights

• RAG in 2026 is described as a combination of an agent harness and a context layer that enables smarter retrieval and task execution.
• Context moves up the stack as MCP servers, skills, and tools become the primary means of guiding agent behavior rather than raw data handling.
• Programs for agents are increasingly defined in English, reducing the dependency on code level definitions for task specification.
• Document context is treated as a universal container, with three core pieces: parsing into token efficient markdown, semantic storage for humans and agents, and repeatable document workflows.
• PDFs and other document formats are designed for display rather than machine interpretation, which motivates advancing OCR and structured extraction techniques.
• ParseBench is used as a public benchmark to measure how well models handle tables, charts, and faithful text extraction across thousands of pages.
• There is a trade off between high accuracy for regulated industries, high speed for indexing large document stores, and cost efficiency for long workflows.
• The platform emphasizes a hybrid approach combining cheaper specialized models with frontier models to balance cost and fidelity.

Questions & Answers

Q: How does the speaker define RAG in 2026 and what are its main components?

RAG in 2026 is described as a decomposition into an agent harness and a context layer. The agent harness handles reasoning and tool use, while the context layer provides access to unstructured data through document parsing, semantic storage, and repeatable workflows. This separation allows retrieval complexity to be managed by the agent, improving task execution and efficiency.

Q: Why is document OCR considered hard according to the talk?

Document OCR is hard because PDFs store text as glyphs with coordinates and tables as line segments, not as functional data structures. Reading order can be arbitrary, especially in multi column layouts, causing machines to misinterpret content. This makes it difficult for agents to reliably extract meaningful information without advanced OCR and structured representations.

Q: What are the three layers of the platform described and what does each do?

The three layers are: parsing, which digitalizes documents into token efficient markdown and metadata; semantic storage, a document management layer for humans and agents to organize and access documents; and repeatable workflows, which encode standardized processes like invoices or KYC to optimize cost and accuracy.

Q: What role does English play in program definition for agents, according to the talk?

English is increasingly used to define runbooks and goals for agents, reducing the reliance on programming languages. This shift enables both technical and non technical users to specify tasks and workflows, helping to scale end to end automation and align agents with business objectives.

Q: What is ParseBench and what purpose does it serve in the discussion?

ParseBench is described as a public benchmark of two thousand human verified pages used to evaluate model performance on tables, charts, and faithfulness. It demonstrates how far current models are from perfect extraction and highlights the remaining gaps in reliably interpreting complex document content.

Q: How does the talk describe the evolution of retrieval in agents?

Retrieval complexity is moving into the agent layer, where the agent learns to search more effectively by choosing the best keywords and search terms. Even with basic retrieval tools, the right queries enable the agent to loop and improve results, reducing the need for hacky retrieval workarounds.

Q: What is the significance of documents as universal containers in the talk?

Documents are framed as universal containers for unstructured context that feed agents. Unlocking this context supports better agent performance as models become more capable and tasks demand richer sources of information, including PDFs, PowerPoints, Word documents, and spreadsheets embedded in organizational data stores.

Q: What future capability does the speaker hint at for autonomous agents?

The speaker suggests a future where agents operate more autonomously on long horizon tasks by leveraging all available context, possibly without explicit English task definitions. They could use a scoring rubric to drive actions and continue looping until goals are achieved, backed by robust contextual data.

Summary & Key Takeaways

• The talk frames RAG in 2026 as a combination of an agent harness and a context layer that offloads retrieval thinking to the agent itself, enabling more autonomous reasoning. It emphasizes the shift from naive retrieval to context aware search terms and task oriented programs.

• Documents are treated as universal containers for unstructured knowledge, with parsing, semantic storage, and repeatable workflows forming a three layer architecture that unlocks document based AI work.

• The presentation highlights OCR challenges, the limitations of PDFs for machine consumption, and the need for accurate token efficient representations to feed both humans and agents while discussing ParseBench and model performance trade offs.