CCloudSuccessTechworld with Randy
Labs / AI Engineer Lab: Local LLMs, RAG, Evals and a Safe Agent
Lab · Beginner–Intermediate · 60–90 min

AI Engineer Lab: Local LLMs, RAG, Evals and a Safe Agent

Run open-source models locally with Ollama, see tokens and temperature for yourself, build a small RAG app over Kubernetes runbooks, write an eval that catches a retrieval failure, and build an AI agent that inspects a real cluster with read-only permissions.

Real recorded runFrom: AI Engineer Roadmap 2026
The output on this page is from our actual run, not edited. Your pod names, IPs and timings will differ; the behaviour won't.

What you'll learn

  • Call a model over plain HTTP and read tokens and latency
  • What temperature really changes
  • Get structured JSON out of a model
  • Embeddings, similarity scores and grounded answers (RAG)
  • Write an eval and find the failure
  • Give an agent tools with least privilege (RBAC)

Before you start

  • Ollama installed (ollama.com)
  • Python 3.10+
  • kind + kubectl for the agent part
  • ~8 GB RAM (the 1.5B model runs on a laptop)

Setup

Pull the models

bash
ollama pull qwen2.5:1.5b
ollama pull nomic-embed-text
# optional, bigger and smarter:
ollama pull qwen2.5:7b

Cluster for the agent part

bash
kind create cluster --name prod-east
kubectl apply -f manifests.yaml
kubectl apply -f checkout-v1.yaml
kubectl apply -f rbac.yaml
kubectl -n shop rollout status deploy/checkout
# break one pod on purpose so the agent has something to find
kubectl apply -f checkout-v2.yaml

Lab files

rag.pyeval.pyagent.pyrbac.yamlmanifests.yamlcheckout-v1.yamlcheckout-v2.yaml

The lab

01Call a model like an engineer

A model is just an HTTP endpoint. Look past the answer at the numbers: prompt tokens, output tokens and seconds. Those are your cost, your context budget and your latency.

randy — zsh
randy@macbook ~ % ollama list
NAME                       ID              SIZE      MODIFIED      
nomic-embed-text:latest    0a109f422b47    274 MB    2 minutes ago    
qwen2.5:7b                 845dbda0ea48    4.7 GB    2 minutes ago    
qwen2.5:1.5b               65ec06548149    986 MB    2 minutes ago
randy — zsh
randy@macbook ~ % curl -s localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "prompt": "In one sentence, what is Kubernetes?", "stream": false, "options": {"temperature": 0}}' | jq '{response, prompt_tokens: .prompt_eval_count, output_tokens: .eval_count, seconds: (.total_duration/1e9)}'
{
  "response": "Kubernetes is an open-source platform for automating the deployment, scaling, and management of containerized applications.",
  "prompt_tokens": 37,
  "output_tokens": 23,
  "seconds": 2.515247066
}

02Temperature

Same prompt, temperature 1.3, twice: two different names. Temperature 0: the same answer every time. Production features usually want low or zero.

randy — zsh
randy@macbook ~ % curl -s localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "prompt": "Give me a name for a Kubernetes monitoring startup. Just the name.", "stream": false, "options": {"temperature": 1.3}}' | jq '{response, prompt_tokens: .prompt_eval_count, output_tokens: .eval_count, seconds: (.total_duration/1e9)}'
{
  "response": "\"KubeLens\"",
  "prompt_tokens": 43,
  "output_tokens": 6,
  "seconds": 0.321724814
}
randy — zsh
randy@macbook ~ % curl -s localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "prompt": "Give me a name for a Kubernetes monitoring startup. Just the name.", "stream": false, "options": {"temperature": 1.3}}' | jq '{response, prompt_tokens: .prompt_eval_count, output_tokens: .eval_count, seconds: (.total_duration/1e9)}'
{
  "response": "KubeWatchers",
  "prompt_tokens": 43,
  "output_tokens": 5,
  "seconds": 0.146572504
}
randy — zsh
randy@macbook ~ % curl -s localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "prompt": "Give me a name for a Kubernetes monitoring startup. Just the name.", "stream": false, "options": {"temperature": 0}}' | jq '{response, prompt_tokens: .prompt_eval_count, output_tokens: .eval_count, seconds: (.total_duration/1e9)}'
{
  "response": "\"KubeLens\"",
  "prompt_tokens": 43,
  "output_tokens": 6,
  "seconds": 0.176854282
}

03Structured output

Ask for JSON and turn on JSON mode, and the model becomes a parser you can call from code.

randy — zsh
randy@macbook ~ % curl -s localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "prompt": "Extract service, error and exit_code as JSON from this log line: \"checkout-8b854d58f-jpsjt  Last State: Terminated  Reason: OOMKilled  Exit Code: 137\"", "stream": false, "options": {"temperature": 0}, "format": "json"}' | jq '{response, prompt_tokens: .prompt_eval_count, output_tokens: .eval_count, seconds: (.total_duration/1e9)}'
{
  "response": "{\n  \"service\": \"checkout\",\n  \"error\": \"OOMKilled\",\n  \"exit_code\": 137\n}",
  "prompt_tokens": 80,
  "output_tokens": 29,
  "seconds": 1.257905664
}

04RAG over runbooks

rag.py embeds six runbooks, finds the closest match to the question by cosine similarity, and answers using only that runbook. Notice it gave four bullets when asked for three. Small models need checking, which is the next section.

randy — zsh
randy@macbook ~ % python3 rag.py
question: my pod keeps dying with exit code 137, what should I do?
top matches:
  0.783  oomkilled.md
  0.624  pending.md
  0.568  dns.md
answer (grounded in oomkilled.md):
- Check kubectl describe pod for Last State OOMKilled to understand the cause.
- Compare pod's memory usage with kubectl top to identify potential leaks.
- Increase the memory limit or fix the underlying issue causing the OOMKilled condition.
- Do not restart the pod; address the root cause to prevent future OOMKilled events.
took 3.3s
randy — zsh
randy@macbook ~ % python3 rag.py "pod stuck pending, scheduler says insufficient cpu"
question: pod stuck pending, scheduler says insufficient cpu
top matches:
  0.897  pending.md
  0.634  oomkilled.md
  0.580  dns.md
answer (grounded in pending.md):
- Check the pod's resource requests and limits.
- Verify if the node has sufficient available resources.
- Ensure there are no node taints conflicting with the pod's tolerations.
- Review the pod's node selector to ensure it matches the node's capabilities.
took 2.2s

05Evals

Six real questions, each with the runbook that should answer it. Five pass, one fails: the image-pull question retrieved the rollback runbook. Now you have a number to improve and a test to stop regressions.

randy — zsh
randy@macbook ~ % python3 eval.py
PASS container killed, exit code 137            -> oomkilled.md
PASS pod never gets scheduled                   -> pending.md
PASS df says 100% but du finds nothing          -> disk-full.md
FAIL ErrImagePull after deploy                  -> rollback.md
PASS cannot resolve external api from pod       -> dns.md
PASS undo the release that broke checkout       -> rollback.md
retrieval accuracy: 5/6

06An agent with least privilege

agent.py gives the model one tool, get_pods, backed by a read-only service account. It finds the crash-looping pod. The can-i checks prove it can list pods but can't delete anything.

randy — zsh
randy@macbook ~ % kubectl -n shop get pods
NAME                       READY   STATUS             RESTARTS      AGE
checkout-5dd967cbb-25gtw   1/1     Running            0             73s
checkout-5dd967cbb-2mwlh   1/1     Running            0             73s
checkout-5dd967cbb-9mlz5   1/1     Running            0             73s
checkout-8b854d58f-4f5zb   0/1     CrashLoopBackOff   3 (22s ago)   71s
storefront                 1/1     Running            0             73s
randy — zsh
randy@macbook ~ % python3 agent.py
user: Is anything broken in the shop namespace?
agent -> tool call: get_pods({"namespace": "shop"})
tool result:
NAME                       READY   STATUS             RESTARTS      AGE
checkout-5dd967cbb-25gtw   1/1     Running            0             88s
checkout-5dd967cbb-2mwlh   1/1     Running            0             88s
checkout-5dd967cbb-9mlz5   1/1     Running            0             88s
checkout-8b854d58f-4f5zb   0/1     CrashLoopBackOff   3 (37s ago)   86s
storefront                 1/1     Running            0             88s
agent: The `checkout` pod is experiencing issues, showing `CrashLoopBackOff`. This indicates that the pod is crashing and restarting repeatedly.
randy — zsh
randy@macbook ~ % kubectl auth can-i list pods -n shop --as=system:serviceaccount:shop:agent-readonly
yes
randy — zsh
randy@macbook ~ % kubectl auth can-i delete pods -n shop --as=system:serviceaccount:shop:agent-readonly
no
randy — zsh
randy@macbook ~ % kubectl auth can-i delete deployments -A --as=system:serviceaccount:shop:agent-readonly
no

Check yourself

8 questions on what happened in this lab. Pick an answer for each, then check.

01A call to qwen2.5:1.5b returns prompt_tokens 37, output_tokens 23 and seconds 2.5. Why should an engineer care about these numbers?
02At temperature 1.3 the same prompt returned "KubeLens" and then "KubeWatchers". At temperature 0 it returned "KubeLens". What should you take from this for a production feature?
03You ask the model to extract service, error and exit_code from a log line and set "format": "json". Why is that more useful than a normal chat answer?
04For the question "my pod keeps dying with exit code 137", rag.py ranked oomkilled.md first at 0.783, even though the question never says "OOMKilled". How did it find the right runbook?
05The RAG prompt asked for three bullet points, but the answer had four. What is the right lesson?
06eval.py scored 5/6. The failure was "ErrImagePull after deploy", which retrieved rollback.md. What is the most likely cause, and a sensible fix?
07The agent is given one tool, get_pods. When it answers "Is anything broken in the shop namespace?", what actually runs the kubectl command?
08kubectl auth can-i shows the agent's service account can list pods but cannot delete pods or deployments. Why is this the right place to enforce safety?
Go deeper

Take this further

The Field Guides cover the whole topic: more labs, a troubleshooting playbook, interview questions and a 30-day plan.

Field Guide · 87 pages · 12 labs

AI Agents for DevOps: Build a Kubernetes Troubleshooter

Build a Python agent that diagnoses real Kubernetes incidents with read-only tools, guardrails, evals and least-privilege RBAC.

$39 CADView guide
Field Guide · 93 pages · 12 labs

Running LLMs on Kubernetes

Serve open LLMs on Kubernetes with confidence: size GPUs with real maths, deploy vLLM, autoscale on the right signals and route traffic smartly.

$39 CADView guide

Or get every guide with All-Access.

Get the next lab first

One real lab every week, straight to your inbox.

One email a week, real labs only. Unsubscribe any time.