CNCF: Before you evaluate an AI SRE, give your existing agent context
The authors ran 50 Kubernetes faults from the SREGym benchmark on a live Amazon EKS cluster, comparing Claude Code with a shell and kubectl against Claude Code with read-only MCP tools from Radar, an open-source Kubernetes UI and MCP server the authors maintain. Both arms reached similar accuracy (46 vs 45 correct), but the context arm reached a correct diagnosis within two minutes on 86% of faults versus 60% with a shell, and used fewer tool calls on average (9.8 vs 16.3). The authors note that the context arm had different permissions from the shell arm, that each arm ran each fault once, and that they maintain the tool being tested.
What changes for platform teams
Platform teams evaluating AI SRE products can first measure whether their existing coding agent with read-only cluster context (current failures, change history, joined resources, filtered logs) diagnoses faults faster, using the published method and a read-only service account. Change history kept beyond Kubernetes event expiry and a read-only identity without Secrets or exec access are the concrete requirements the authors recommend checking.
Read the original at CNCF Blog ↗
written by claude-haiku-5-5, prompt 7cbe154, checked by publish rules