How to Build Andrej Karpathy’s LLM Knowledge Base with Claude Code and Markdown

TL;DR

You can build Andrej Karpathy’s LLM knowledge-base workflow in about five minutes by giving raw documents or transcripts to Claude Code, which organizes them into linked markdown wiki pages that Obsidian can display. The demonstrated system automatically structured 36 YouTube transcripts and supports a research collection of about 100 articles and half a million words without a vector database. Read on for the folder structure, ingestion process, and querying workflow.

Transcript

What you're looking at right here is 36 of my most recent YouTube videos organized into an actual knowledge system that makes sense. And in today's video, I'm going to show you how you can set this up in 5 minutes. It's super super easy. You can see here how we have these different nodes and different patterns emerging. And as we zoom in, we can se... Read More

Key Insights

• Karpathy's method utilizes LLMs and markdown files to create efficient knowledge bases.
• This approach eliminates the need for complex infrastructure like vector databases.
• Obsidian is used as an IDE to visualize and organize markdown files.
• The LLM automatically organizes raw data into a structured wiki, allowing for easy querying.
• Knowledge bases created this way are persistent, unlike ephemeral AI chat interactions.
• The setup process is simple, taking roughly five minutes to establish a basic system.
• Token efficiency is significantly improved, reducing costs by up to 95% in some cases.
• This method is suitable for small to medium-scale projects but may not scale for large enterprises.

Questions & Answers

Q: How do you build an LLM knowledge base with Claude Code and markdown?

Create a vault containing a raw folder and a wiki folder, then place source documents such as PDFs or YouTube transcripts in the raw folder. Ask Claude Code to compare and organize the material into linked markdown pages inside the wiki folder. It can also maintain an index, brief document summaries, and an operation log.

Q: How long does the LLM knowledge-base setup take?

The presenter says the basic system can be set up in about five minutes. The setup is simply a folder-based vault of markdown files rather than a vector database, embeddings pipeline, or other complex infrastructure.

Q: What does Claude Code do with the raw source material?

Claude Code reads the raw documents, compares their contents, and places the information into appropriate wiki pages with relationships and backlinks. In the demonstrated YouTube system, it organized 36 recent video transcripts without manual relationship building. The resulting pages include explanations, takeaways, tags, links, and connections to related tools and techniques.

Q: What folder structure does the markdown knowledge base use?

The example vault is called “my wiki” and contains a raw folder for source material and a wiki folder for pages created by the LLM. The wiki also contains an index for navigating tools, techniques, concepts, sources, people, and comparisons, plus a log that records operation history.

Q: What role does Obsidian play in Andrej Karpathy’s workflow?

Obsidian serves as the IDE and visual interface for the markdown files. It lets users inspect wiki pages, follow backlinks, and view emerging relationships as connected nodes, while the underlying knowledge base remains a collection of files in folders.

Q: Can the knowledge base answer questions across all the documents?

Yes. During the Q&A phase, the LLM can search through the wiki and use related pages to answer questions about the research or YouTube material efficiently. It can also identify gaps in a node or relationship and conduct research to fill those gaps.

Q: Does this method require a vector database or RAG infrastructure?

No fancy vector database, embeddings system, or complex infrastructure is required for the demonstrated small-scale workflow. The transcript says the LLM can maintain index files and brief summaries, then read the important related data directly from the markdown wiki.

Q: What scale does Karpathy’s LLM knowledge base handle in the example?

Karpathy’s example covers about 100 articles and roughly half a million words. The presenter separately demonstrates a knowledge system built from 36 recent YouTube videos and notes that the file-based approach works at this small scale.

Summary & Key Takeaways

• Andrej Karpathy's method leverages LLMs and markdown files to build knowledge bases without complex infrastructure. Using tools like Claude Code and Obsidian, users can efficiently organize and query information. This method offers persistent knowledge retention, unlike traditional AI chat interactions, and is cost-effective due to improved token efficiency.

• The setup involves creating markdown files that the LLM organizes into a structured wiki. Obsidian serves as the front end for visualizing and managing these files. This approach is accessible, requiring only basic markdown files, and is particularly useful for small to medium-scale projects.

• Karpathy's approach is gaining traction for its simplicity and efficiency. By eliminating the need for vector databases, it offers a straightforward way to maintain and query knowledge bases. However, it may not be suitable for large-scale enterprise applications due to scalability limitations.