AI Agents for DevOps: Build a Kubernetes Troubleshooter

This is for you if

• You run Kubernetes and want to know what an ops agent really is in code
• You write some Python and know kubectl, Deployments, Services and RBAC
• You are interviewing for SRE, platform or AI-engineering roles
• You will be asked to review or approve someone else's agent

What's inside

• A complete ~500-line agent: tool-calling loop on the OpenAI-compatible API
• Four read-only kubectl tools with allowlists, validation, redaction and truncation
• Running the same agent on Ollama locally and on a hosted API
• Structured outputs: a Pydantic diagnosis with self-correcting validation
• Prompt injection defences, hard vs soft guardrails, and an injection test case
• Six seeded incidents and an eval harness with pass rate, p95 latency and tokens
• Token and latency arithmetic, plus the levers that actually cut cost
• In-cluster Job with a get/list-only ClusterRole and per-namespace RoleBindings
• Human-approved write actions with server-side dry runs and an audit log

After this guide you can

• Explain and build an LLM tool-calling loop without a framework
• Design agent tools and RBAC that stay safe when the model is fooled
• Evaluate an agent on seeded incidents and compare models with real numbers
• Estimate and reduce the cost and latency of agent runs
• Deploy the agent in-cluster with least privilege and a human approval step

Contents

1. The mental model: what an agent actually isp. 5
2. Tool calling from first principlesp. 9
3. Read-only kubectl tools that cannot hurt youp. 14
4. The agent loop and structured outputsp. 26
5. Guardrails: hard limits, soft limits and prompt injectionp. 39
6. Evaluating the agent on seeded incidentsp. 43
7. Cost and latency: where the tokens gop. 51
8. Deploying in-cluster with least-privilege RBACp. 55
9. Human in the loop for write actionsp. 65
10. Troubleshooting playbookp. 70
11. Production checklist and design decisionsp. 73
12. Interview questionsp. 76
13. Cheat sheet and 30-day practice planp. 83