The wrong route to AI SRE

I was exploring how to build AI SRE agents for a few weeks. Did the research through Claude, and it pointed me to this paper. So I started skimming it through and started looking at the solution from the lens of the paper. The paper was interesting and the logic was math heavy, so I spent more time refreshing probabilistic concepts through Claude, building visualisation and intuition. I tried different detection methods, naive shift, robust z-scores, posterior predictive checks, lead-time analysis, fusion variants and implemented, benchmarked and documented on the RCAEval dataset. I didn’t do the actual implementation myself obviously. Claude Code did most of the work tbh. The results weren’t too bad and the process looked logical. But the moment I tested it on real incidents, the results were disappointing. The algorithm would attribute root cause to internal services which were not even important. It also failed to effectively use correlations and traces. The problem is figuring out the right numbers is very difficult when there are a lot of services. The algorithm is effectively doing two things: 1. which service moved first and 2. which service deviated the most. And services which are mostly constant and move little can have huge implications, since the variance is very low before and will have a huge variance ratio.

Writing algorithms to solve RCA was the wrong route for a few reasons. The agents are more than capable of implementing them on their own. Rather than manually hard coding methodologies like the USE method or the RED method (Chapter 2, Systems Performance), agents are able to do that themselves and more. An agent that investigates like an engineer worked better.

What RCA actually is

The pipeline framing treats root-cause analysis as a detection problem with metrics as input, ranked suspects as output. If I look at an incident from the on-call engineer’s point of view, they do three things, apart from the incident management and coordination:

1. Try to recall if a similar incident has taken place before, and try to implement the runbook.
2. Look at dashboards and check if they can see anything different, aka anomaly detection by visualisation and common sense.
3. Make a hypothesis, and decide what next and where to look.

So in essence, there is no fixed path to RCA. One has to figure it out on the go, based on the results of the hypothesis. This was the major gap in my v0 implementation. And this is the first time where the problem is agentic. All the other times, the problem was mostly workflows, and to build them as agents is force fitting it just because agentic is cool.

So I started testing agents within my own Claude Code environments. And when I looked at the traces, the agent already implemented the methods which I was hard coding myself.

Where the statistics went

The current system is an agent in a coding environment with logged query access to telemetry, driven by an investigation skill. The statistical learnings now go into a principles.md file, the general directions for the agent to look at, like:

• use magnitude as a filter, and not to rank services
• timing gives direction only
• topology and traces determine the timeline

And the minor learnings, like median absolute deviation and robust scoring, become part of these principles, to be used by the agent only as guidance.

Learning Sutton’s bitter lesson by making mistakes while building an AI SRE agent on RCAEval

The learning from this is to rely more and more on agent intelligence, use domain knowledge to steer, provide principles as guidance, and not focus too much on implementation. And this has become my default experiment pattern: I just let the agent solve the problem, look at traces to build intuition on how the agent operates, fine-tune the principles.md file, create a skill from it, and then run on the eval dataset before even thinking of the algorithm and the implementation.