How to Build an LLM Knowledge Base from Notes

TL;DR

Capture as many raw thoughts as possible, preferably through fast voice dictation, then let agents organize the resulting Markdown files. A practical pipeline can enrich notes with controlled tags, timestamps, researched sources, and backlinks, generate a browsable personal wiki, synchronize updates on a schedule, and build an HTML graph that reveals relationships and gaps across your thinking.

Transcript

[music] All right. Are we ready to go? Okay, we're ready to go. Are y'all ready to go? >> We ready? All right. Let's do this thing. Uh, all right. Well, hello everyone. I'm Ben Holmes, developer relations lead at Warp. Uh, you might have heard of Warp as a terminal that's really nice to use for you and your coding agents. You may have also heard of... Read More

Key Insights

• Raw material is the foundation of an LLM knowledge base because agents need a substantial collection of thoughts before they can generate useful links, wikis, and visualizations. Notes can remain rambling and imperfect during capture because organization happens in later processing passes.
• Voice dictation is presented as the fastest way to capture ideas because ordinary speech runs at roughly 200 words per minute. It allows someone to record detailed reactions immediately after a podcast, meeting, research passage, or other experience without first designing a polished note structure.
• Local transcription tools are available for capturing speech without requiring a continuing subscription. Handy is described as open source and powered by a local model, while Voice Ink offers hotkey and mobile capture, on-device processing, and formatted sentences and paragraphs after transcription.
• Markdown files are a practical storage format because people and agents can both navigate and modify them. The notes can be viewed through Hubble, an application built by Holmes, or through other tools such as Warp, Obsidian, and general Markdown viewers.
• An enrich-note skill is responsible for converting raw notes into organized knowledge. It adds a processing timestamp, assigns topic tags, researches the original source with web tools, and appends backlinks to related files discovered through keyword and file searches.
• A fixed tag reference list limits inconsistent categorization across repeated agent runs. The agent is instructed to reuse existing tags and remain reluctant to invent new ones, while retaining permission to add a category when it identifies a meaningful recurring pattern.
• Backlinks create a navigable web of personal ideas by connecting notes with related subjects. These links make it possible to move through earlier thoughts like a personal Wikipedia rabbit hole instead of manually searching scattered entries saved weeks apart.
• Automated enrichment can run on a schedule in a cloud sandbox using the Obsidian headless CLI. The system synchronizes Markdown files, finds notes without enrichment timestamps, processes them, regenerates browsable material, and syncs the results back for a refreshed personal knowledge collection.

Questions & Answers

Q: How do you build an LLM knowledge base from personal notes?

Start by collecting raw thoughts in Markdown without demanding perfect formatting. Capture meeting transcripts, reactions to podcasts, research notes, and observations from books. Then use an agent skill to timestamp each processed file, apply tags from a controlled reference list, research its source, and add backlinks. The enriched collection can subsequently become a browsable wiki and an HTML graph of related ideas.

Q: Why should voice dictation be used for capturing knowledge?

Voice dictation makes it easier to collect enough raw material for later agent processing. Holmes says talking runs at roughly 200 words per minute, so speaking can be faster than typing unless someone is an exceptionally fast typist. Dictation also encourages immediate, detailed capture after meetings, podcasts, or reading, without requiring the speaker to stop and format every thought as polished bullet points.

Q: What tools can transcribe notes locally on a device?

Handy and Voice Ink are presented as local options for voice dictation. Handy is an open-source tool that uses a local model and keeps processing on the device. Voice Ink is described as having a roughly $20 lifetime fee for application updates, along with a computer hotkey and a mobile app. It converts continuous speech into text with punctuation and paragraph breaks.

Q: What should an enrich-note agent skill add to each file?

An enrich-note skill should add a timestamp showing when the file was processed, select suitable tags from an established reference list, investigate the original source with web tools, and append links to related notes. Keyword and file searches can identify those relationships. The timestamp also helps later automation distinguish files that have already been enriched from notes that still require processing.

Q: How do controlled tags improve an AI knowledge base?

Controlled tags give agents a concrete set of categories to consult instead of allowing every processing run to invent different labels. Holmes stores his tags in a reference folder and instructs the agent to be reluctant to add new ones. The system may still extend the list when it detects a meaningful pattern, but reuse remains the default, producing a more consistent and searchable notebook.

Q: How do backlinks make personal notes easier to navigate?

Backlinks connect files that discuss related topics, even when those notes were captured at different times. An agent can discover candidates through keyword and file searches, then place links at the bottom of the enriched note. As more material is processed, the network becomes tighter and lets the reader follow paths among personal ideas much like moving through related entries in Wikipedia.

Q: How can agents generate a personal wiki from notes?

After notes have been enriched with tags, sources, and backlinks, an agent can combine that material into a more browsable wiki. The described approach generates entries for the people, concepts, organizations, and sources contained in the notebook, using a Karpathy gist as part of the wiki-generation approach. The result supports clicking through areas of research and personal interest rather than reading only isolated transcripts.

Q: How can an LLM knowledge base update automatically?

The workflow can run on a schedule inside a cloud sandbox. The Obsidian headless CLI synchronizes the Markdown collection into that environment, and an agent finds notes that do not yet have enrichment timestamps. It processes those files, updates the wiki, and synchronizes the results back. Holmes compares the refreshed result to a personal daily paper that is ready when the user returns.

Summary & Key Takeaways

• The workflow begins with abundant raw material rather than careful organization. Voice dictation can capture roughly 200 words per minute, making it useful for recording reactions to podcasts, meetings, research passages, or books. Local tools such as Handy and Voice Ink can transcribe speech on device and format it into readable sentences and paragraphs.

• An enrich-note agent skill turns unstructured Markdown into a connected collection. It timestamps processed files, selects categories from a fixed reference list, researches original sources with web tools, and discovers related notes through keyword searches. Being reluctant to create new tags prevents uncontrolled category growth while still allowing genuinely recurring patterns to become categories.

• The enriched collection can support generated wiki entries for people, concepts, organizations, and sources. A scheduled cloud sandbox can use the Obsidian headless CLI to synchronize Markdown, enrich unstamped notes, regenerate the wiki, and sync changes back. Agents can also create an HTML and Tailwind graph that exposes connections and underdeveloped areas of thought.