How Can AI Agents Diagnose DevOps Incidents?

TL;DR

AI agents can reduce root cause analysis from hours to 2–4 minutes by traversing dependency maps and connecting telemetry with problematic code changes. This approach becomes increasingly important as AI-generated code enters mission-critical systems, where failures often emerge from unforeseen interactions and human engineers may lack the context needed to diagnose code they did not write.

Transcript

If you're in product, if you're in design, if you're in core engineering, you constantly have to make bets as to where AI is going to be six months from now and you're willing to re-evaluate everything six months from now. The good news is that it's only going only to get better. So, it's not like your product's going to get worse 6 months from now... Read More

Key Insights

• DevOps and SRE are expected to continue existing, but their work could shift from repeatedly handling severe incidents and daily alerts toward creative, long-term infrastructure planning as AI agents assume more urgent and repetitive operational responsibilities.
• Production incidents resemble medical emergencies because a high-severity failure demands immediate attention, while recurring alerts and deployment checks resemble chronic problems that prevent teams from focusing on long-term infrastructure health and optimization.
• AI-generated code creates a maintenance challenge because code can pass local unit tests while still failing when separate components interact in unforeseen ways inside a large enterprise system.
• Root cause analysis is the process of tracing a production incident to its underlying cause so teams can choose an immediate rollback or hot fix and later develop a durable correction.
• Incident response often escalates through several organizational layers before DevOps or SRE teams determine severity, create an incident channel, and bring together 30 to 50 people to investigate the failure.
• Observability is centered on creating, storing, and visualizing MELT data, meaning metrics, events, logs, and traces, so engineers can inspect different parts of a software system during troubleshooting.
• Traversal's agents perform root cause analysis in 2–4 minutes by systematically traversing complex dependency maps to identify the relevant logs and code changes associated with a production failure.
• Traversal's architecture was designed around the expectation that reasoning models would improve, allowing stronger models to contribute more effectively to the diagnostic process as their reasoning capabilities advanced.

Questions & Answers

Q: How can AI agents diagnose production incidents faster?

AI agents can diagnose incidents by systematically traversing complex dependency maps and searching for the logs and code changes most closely associated with a failure. Traversal reports that its troubleshooting agents can perform root cause analysis in 2–4 minutes, compared with investigations that typically occupy teams of engineers for hours in incident channels and war rooms.

Q: What is root cause analysis in DevOps?

Root cause analysis is the investigation used to identify the underlying reason a production system failed. It sits within an incident lifecycle that begins with a customer problem, moves through support and engineering escalation, and may become a formal incident. Once the cause is identified, teams can roll back a change, apply a hot fix, and plan a longer-term correction.

Q: Why does AI-generated code make production debugging harder?

AI-generated code can pass local unit tests while still creating failures when it interacts with other components in a large enterprise system. Those interactions may be difficult to foresee. Debugging becomes harder because engineers may not have written the code and therefore lack its original context, making AI-powered software maintenance increasingly important for reliable mission-critical systems.

Q: Will AI eliminate DevOps and SRE jobs?

DevOps and SRE are expected to remain, but their responsibilities could look fundamentally different. AI agents may take over high-severity troubleshooting, repetitive alerts, and routine deployment checks. Human engineers could then spend more time on creative planning, including deciding how infrastructure should evolve over the next year or five years and where current investments should be made.

Q: What happens during a typical production incident?

A customer first encounters a malfunction, or an existing monitoring process records it. Customer support determines whether it is a user error or a genuine issue, then escalates it through engineering groups. DevOps or SRE teams assess impact, severity, and immediacy. A serious incident may create a Slack channel with 30 to 50 participants who investigate, mitigate, and plan repairs.

Q: What is the relationship between observability and root cause analysis?

Observability creates, stores, and visualizes telemetry known as MELT data: metrics, events, logs, and traces. Dashboards and other views give engineers visibility into different parts of a system. Root cause analysis goes beyond viewing that data by connecting evidence across the system to determine which event, dependency, log, or code change actually produced the failure.

Q: Why are healthcare analogies useful for understanding DevOps?

Healthcare analogies distinguish three levels of reliability work. A high-severity incident resembles a heart attack that requires immediate action. Constant alerts and deployment checks resemble a chronic condition that consumes daily attention. Long-term infrastructure planning resembles improving sleep or nutrition, where the objective is sustained health rather than emergency treatment or continuous symptom management.

Q: How should companies plan AI diagnostic products as models improve?

Companies can make architectural bets based on the expectation that reasoning models will improve, while remaining willing to reevaluate those bets after six months. Traversal designed its system so reasoning models could play a central role and benefit from stronger capabilities. The founders say that this decision paid dividends as model reasoning improved and aligned with their architecture.

Summary & Key Takeaways

• Traversal applies AI agents to production troubleshooting and root cause analysis. Its agents examine complex dependency maps to locate important logs and problematic code changes, reducing investigations that traditionally consume hours of engineering time to 2–4 minutes in reported deployments. The goal is to automate urgent and repetitive reliability work.

• Traditional incident response commonly escalates from customers and support teams to DevOps or SRE specialists. A high-severity incident can produce a Slack channel containing 30 to 50 people, followed by a search for an experienced engineer who can identify the cause, coordinate a rollback or hot fix, and guide longer-term remediation.

• AI-assisted coding creates two different reliability contexts. Disposable experiments may tolerate failures, but mission-critical payments, financial, security, and streaming systems cannot. Locally successful AI-generated code may still fail through unexpected system interactions. AI-powered maintenance is therefore presented as necessary for diagnosing software that engineers may not personally understand or have written.