How to Build an Agentic OS for AI Workflows

TL;DR

Build an agentic OS around three connected layers: a custom visual interface, an Obsidian-based memory system, and reusable skills that become automations. Route voice or text requests by complexity, using direct commands for simple actions, small models for basic reasoning, and Claude Code or Codex for complex work, while logging conversations and outputs back into Obsidian.

Transcript

you need an Aentic OS, but it's not for the reasons you think. It's not because you need fancy dashboards like this or visual interfaces like this one, even though I'll show you how to create them. It's instead about building a custom system that takes the best of all the AI tools at your disposal, whether that's GPT6 Astra or Claude Opus 5.5 or Je... Read More

Key Insights

• An agentic OS is a three-layer system composed of a visual interface, an Obsidian-based memory structure, and a reusable skill architecture. Its purpose is to combine available AI tools around a person's actual work rather than depend on one terminal interface.
• The visual layer is fully customizable and can display calendars, social accounts, audience information, research, skills, reports, and automations. It can operate as a browser application or an Obsidian plugin while retaining terminal access and support for Claude Code or Codex.
• The memory layer is based on meaningful files and folders rather than an attractive knowledge graph. Its practical goal is to help both the user and the AI system locate accurate answers quickly, even when the collection contains thousands of files.
• The skill architecture is the operational backbone of the system. Repeated daily and weekly activities can be defined as skills, converted into automations, and used to produce reports, graphics, and other deliverables that return to the agentic OS.
• The voice pipeline works locally by using Whisper to transcribe spoken requests and Kokoro to generate spoken responses. A bridge connects the interface to this pipeline, records activity in Obsidian, and sends the transcribed request onward for classification.
• Tier-one requests are direct actions that require no AI reasoning, such as opening a morning intelligence brief or terminal. Jev classifies these requests so the system can execute them directly, producing responses within hundreds of a millisecond according to the presentation.
• Tier-two requests require limited reasoning and are routed to the smallest available model, identified as Haiku or Luna depending on the system version. The presenter also states that these models could be replaced with a model running entirely on the user's machine.
• Tier-three requests are complex work products, such as creating a visual comparison or explainer. The system opens Claude Code or Codex in an embedded terminal, lets the user interact with that environment, and logs the resulting conversations and deliverables back into Obsidian.
• More videos from this channel:
  • How to boost Claude Code search with last 30 days skill
  • How Does Claude Opus 5.5 Compare on Cost?
  • How to Create Videos with Claude Code and Remotion
• More from this channel: Chase AI

Questions & Answers

Q: What is an agentic OS for AI workflows?

An agentic OS is a custom system that combines multiple AI tools around the user's real work. The described design has three layers: a visual interface for accessing information and actions, an Obsidian-based memory system for storing and retrieving context, and a skill architecture that converts recurring work into reusable skills, automations, reports, graphics, and other outputs.

Q: What are the three layers of an agentic OS?

The three layers are the visual layer, the memory layer, and the skill layer. The visual layer provides customized access to reports, calendars, research, terminals, and automations. The memory layer organizes information through Obsidian files and folders. The skill layer defines repeatable work, supports automation, and produces deliverables that are fed back into the system.

Q: How does an agentic OS route requests to different AI tools?

The system uses Jev to classify each request into one of three tiers. Direct actions can run without an AI model, basic questions can go to a smaller model such as Haiku or Luna, and complex work can open Claude Code or Codex in a terminal. This structure matches each request with the amount of reasoning and tooling it needs.

Q: How does local voice control work in the agentic OS?

Spoken input first reaches an Obsidian plugin or web interface and then passes through a bridge. Whisper runs locally and converts the audio request into text. Kokoro, another local open-source model, generates the spoken reply. The bridge also records activity in Obsidian and forwards the transcription to Jev so the request can be classified and routed.

Q: Why is Jev used as a model router?

Jev is used because it can classify fuzzy natural-language requests and return numerical probabilities rather than conversational sentences. The presentation describes it as suitable for deciding which route a request should take. It is also claimed to perform this classification about 200 times faster and cheaper than large language models, making it useful for frequent routing decisions.

Q: What is the difference between tier-one, tier-two, and tier-three requests?

Tier-one requests are simple commands, such as opening a report or terminal, and do not invoke an AI model. Tier-two requests need some reasoning but remain non-complex, so they use a small model such as Haiku or Luna. Tier-three requests involve substantial work, such as creating an explainer, and are routed to Claude Code or Codex in a terminal.

Q: How does Obsidian function as the memory layer?

Obsidian stores conversations, deliverables, and other system activity in a file and folder structure that is understandable to both the user and the agentic OS. The emphasis is not on knowledge graphs or a conventional retrieval-augmented generation system. Instead, the structure is intended to support fast, accurate answers across collections containing thousands of files.

Q: How can recurring work become skills and automations?

Recurring activities performed every day or week can first be documented as reusable skills. Those skills can then be turned into automations that generate concrete outputs, including reports and graphics. The outputs are returned to the agentic OS, where the visual layer can display them and the Obsidian memory layer can preserve them alongside related conversations and context.

Summary & Key Takeaways

• An agentic OS combines a customizable interface, a structured memory layer, and a skill architecture. The interface surfaces reports, calendars, research, terminals, automations, and audience data. It can run as a browser application or an Obsidian plugin, with Claude Code or Codex handling work that requires terminal access.

• The routing architecture begins with voice or text input. Local Whisper converts speech into text, while the open-source Kokoro model produces spoken responses. A bridge records activity in Obsidian and passes requests to Jev, which classifies them into direct commands, lightweight reasoning tasks, or complex terminal-based work.

• The skill layer turns recurring daily or weekly work into reusable procedures and then automations. Those processes create reports, graphics, and other deliverables that feed back into the system. Obsidian stores conversations and outputs in a human-readable file and folder structure designed to support accurate retrieval across thousands of files.