How to Give Hermes a Self-Improving Knowledge Base

TL;DR

Connect Hermes to an Obsidian-based LLM wiki so it can use external knowledge from documents, meetings, research, and other sources alongside its conversational memory. An injection workflow can process new files, create source pages, update related pages, check linkages, and flag contradictions, allowing the knowledge base to become more useful as information is added.

Transcript

Hermes has the best memory of any AI agent. It remembers your conversations and gets smarter the more that you use it. But, there's one thing it can't do. And once you fix it, you give Hermes superpowers and unlock completely new capabilities. So, in this video, I'll show you exactly how to fix that [music] to give Hermes a self-improving knowledge... Read More

Key Insights

• Hermes memory is a conversational loop that extracts durable facts into files such as memory.md or user.md, then recalls or searches those facts during later conversations. This allows the agent to understand the user better as repeated interactions add more personal context.
• Hermes has a blind spot because its normal memory only knows what the user has explicitly shared. Email threads, meeting notes, calls, documents, research, NotebookLM material, and useful videos remain outside that conversational memory unless they are deliberately made accessible.
• An LLM wiki is an external knowledge base that rewrites and expands itself as new material arrives. It can organize expert commentary, research, transcripts, and historical context into a self-referential collection that the agent can search when answering questions.
• The injection workflow is responsible for maintaining the wiki when a file is added. It reads and discusses the material, writes a source page, updates affected pages, checks relevant linkages, and flags contradictions between new information and existing knowledge.
• The combined system gives Hermes two complementary forms of context. Hermes memory contains knowledge about the user, while the wiki contains knowledge about the user's world, including meetings, external expertise, documents, and research that can support better-informed responses.
• The wiki can be created by giving a capable language model the referenced LLM wiki instructions and asking it to build the required desktop folder structure. The same setup can be performed through Hermes, Claude, or another model capable of following those instructions.
• A local Obsidian vault becomes available to Hermes by supplying its file location and explaining its purpose. Hermes should be told that the vault contains external information it may query during conversations, particularly for decisions involving meetings, notes, research, or strategy.
• A dedicated LLM wiki or Obsidian skill can define when Hermes should consult the external knowledge base. The skill can be tested with practical requests, such as asking for content-introduction advice drawn from previously ingested YouTube strategy transcripts.

Questions & Answers

Q: How does Hermes memory work?

Hermes uses a conversational memory loop. During a conversation, it extracts durable information and writes notes into files such as memory.md or user.md. In later conversations, it can recall or search those stored facts before answering. This process helps Hermes understand the user more deeply over time, but it primarily covers information that has entered through direct interactions.

Q: Why does Hermes need an external knowledge base?

Hermes needs an external knowledge base because its conversational memory only contains information the user has shared with it. Important context may instead live in inbox threads, calls, meeting notes, documents, research, NotebookLM, or useful videos. Connecting those sources gives Hermes access to knowledge about the user's wider work and environment, not just personal conversational history.

Q: What is an LLM wiki for Hermes?

An LLM wiki is an external, self-improving knowledge collection that Hermes can query during conversations. It can hold expert commentary, research, transcripts, meeting information, and historical context. As new material is added, the system checks relationships, updates relevant pages, and identifies contradictions, creating an interconnected reference that becomes more useful as it grows.

Q: How does the wiki injection workflow process new files?

When a new file enters the wiki, the injection workflow reads it, discusses its contents, and writes a source page. It then updates pages affected by the new information, examines relevant linkages, and flags contradictions with existing material. This maintenance process integrates new knowledge into the wider collection instead of leaving each source as an isolated document.

Q: How do you create the Obsidian wiki folder structure?

Open the referenced LLM wiki instructions, copy their URL, and give it to a language model such as Hermes or Claude. Ask the model to familiarize itself with the concept and create a desktop folder containing the required structure. Once completed, the resulting Obsidian wiki can hold source material, related pages, and instructions for processing future additions.

Q: How do you connect an Obsidian vault to Hermes?

Find the local file location of the Obsidian vault, then provide that location to Hermes through the dashboard connection process described in the transcript. Explain that the vault contains external knowledge, including meeting transcripts and expert insights, which Hermes should query when relevant. This makes the wiki a reference source alongside Hermes's existing conversational memory.

Q: How should a Hermes skill use the LLM wiki?

Create a persona or skill dedicated to the LLM wiki and define the situations in which it should be used. The instructions can tell Hermes to consult the vault for strategy questions, meeting context, notes, and other external knowledge. A system prompt can formalize this behavior, and a practical query can verify that Hermes retrieves information from the wiki.

Q: How can you test whether Hermes is using the wiki?

Add useful source material to the wiki, such as transcripts about YouTube content strategy, then ask Hermes a question that depends on that material. The example in the transcript requests several tips for creating a better introduction. If Hermes searches the connected Obsidian wiki and returns advice based on the stored sources, the connection and retrieval workflow are functioning.

Summary & Key Takeaways

• Hermes records durable details from conversations in memory.md or user.md and can recall or search those details before responding. Its main limitation is that this conversational memory only contains information the user has shared, leaving external material such as inbox threads, calls, documents, research, meeting notes, and videos outside its normal context.

• The proposed solution is an Obsidian-based LLM wiki inspired by Andrej Karpathy's principles. Its injection workflow reads each new file, discusses its contents, writes a source page, updates affected pages, checks relationships, and flags contradictions. This creates an expanding, self-referential knowledge base that becomes stronger as more information is ingested.

• After creating the wiki folder structure, the user connects its local vault location to Hermes and defines when the agent should consult it. A dedicated persona or skill can direct Hermes to query the wiki for strategy, meetings, notes, or expert insights, combining knowledge about the user with knowledge about the user's wider world.