How to understand AI agents and levels of agency

TL;DR

AI agents center on an LLM paired with tools and autonomy to act in the real world. There are three levels of agency: Level 1 combines an LLM with tools, Level 2 uses LLM workflows, and Level 3 places an LLM in a loop for ongoing interaction. Tools include web search, code interpreters, memory, and API access, enabling real-world feedback and actions.

Transcript

hey everyone I'm Shaw this is the first video in a larger series on AI agents here I'll start by discussing what an AI agent actually is and why we should care about them then I'll review three common types of agents at various levels of agency 2025 is said to be the year of AI agents yet for many people that I've talked to it's not entirely clear ... Read More

Key Insights

• An AI agent always centers on an LLM, represented by a robot emoji, and is augmented by tools and software to extend its capabilities.
• Tool use is essential for agents, enabling web search, a code interpreter, memory, API calls, and more to provide real world context and actionable outcomes.
• Autonomy refers to the model’s ability to plan, reflect, and adapt outputs based on feedback, reducing the need for explicit step by step instructions.
• Level 1 agents combine an LLM with tools, but often rely on a single model and a single thread of information, limiting scalability for complex tasks.
• Level 2 agents use predefined LLM workflows, which sequence tasks and involve one or more LLMs to achieve goals.
• Level 3 agents place the LLM in a loop, enabling sustained interaction with tools and the environment for more complex, ongoing tasks.
• The practical value of agents lies in bridging imagination and reality through tool use and feedback, addressing hallucinations and enabling real world results.
• Definitions of agents vary across organizations, but the practical view frames agency as a spectrum from no agency to high autonomy.

Questions & Answers

Q: A real search query question about AI agents, What is an AI agent and why should I care?

An AI agent is a system that centers on a large language model and augments it with tools to interact with the real world, perform tasks, and receive feedback. This combination enables more capable, autonomous behavior than a vanilla chatbot. The value comes from enabling real world actions, decision making, and problem solving across diverse tasks.

Q: What is the role of tools in AI agents?

Tools expand an LLM’s capabilities beyond text generation by enabling actions such as web search, running code in a Python interpreter, accessing memory, and making API calls. These tools provide real world context, up-to-date information, and the ability to execute tasks, which helps agents produce correct, useful outcomes rather than only plausible text.

Q: What does autonomy mean in the context of AI agents?

Autonomy refers to an agent’s ability to direct its own processes, reflect on outputs, and adjust its approach based on feedback rather than following strictly scripted instructions. This capability, often linked to planning and reasoning, allows agents to tackle increasingly complex tasks with less manual prompting.

Q: What is Level 1 AI in the framework discussed?

Level 1 AI combines an LLM with tools. It involves using tools like web search or a code interpreter to augment the LLM, enabling it to access fresh information and perform computations. However, it may still rely on a single model and a linear execution flow, limiting multi-task capabilities.

Q: What is Level 2 AI in the framework discussed?

Level 2 AI uses LLM workflows, which are predefined sequences of steps that incorporate one or more LLMs to accomplish tasks. This structure allows more complex, multi-stage tasks to be handled more reliably than a single prompt, and is a common approach in modern agentic systems.

Q: What is Level 3 AI in the framework discussed?

Level 3 AI embeds the LLM in a loop, enabling ongoing interaction with tools and the environment. This level supports continuous decision making, real-time feedback, and sustained problem solving for highly complex tasks that require adaptive behavior over time.

Q: Why are agents considered more valuable than basic chatbots?

Agents are more valuable because they can interact with reality through tools, fetch current information, perform actions, and reason over time. This ability to act rather than just respond makes them capable of solving real world problems and delivering tangible outcomes rather than offering only static text.

Q: How do different organizations define AI agents and why does that matter?

Different organizations define AI agents with varying emphasis on tools, planning, and autonomy. OpenAI focuses on tool usage, Hugging Face emphasizes planning, and Anthropic highlights autonomous control. Despite these differences, the practical view sees agency as a spectrum and focuses on the combination of LLMs, tools, and feedback to achieve useful results.

Summary & Key Takeaways

• AI agents center on an LLM and surrounding components that enable execution and real world feedback. The video explains why agents matter, how they differ from simple chatbots, and how tool use and autonomy expand capabilities.

• Three levels of agency are described. Level 1 is LLM plus tools, Level 2 involves predefined LLM workflows, and Level 3 embeds the LLM in a loop for continual interaction and decision making.

• The talk surveys common definitions from OpenAI, Hugging Face, and Anthropic, emphasizing tool usage, planning, reflection, and the spectrum of agency from basic to highly autonomous systems.