AI Engineer Roadmap 2026: What to Learn and What to Skip

There are two very different AI careers, and most roadmaps mix them up.

One is research. Linear algebra for six months, then you train a neural network from scratch. That's a great path if you want to work at a research lab. The other is AI engineering: taking models that already exist and turning them into products that are fast, cheap, safe and reliable. That's an engineering job, and it's what most "AI engineer" postings actually describe.

This roadmap is for the second one. It has seven stages, and we built and ran every one of them on real infrastructure: small open-source models served locally with Ollama, and a real Kubernetes cluster for the agent part. The models are small on purpose. You'll see them make mistakes, and seeing them fail is half the lesson.

Stage 1: Foundations (and what to skip)

Boring is what gets you hired here.

• Python, properly. Functions, classes, virtual environments, async, calling APIs. Almost every AI tool speaks Python first.
• APIs and JSON. To your code, a model is an HTTP endpoint that takes text and returns text. You need to be comfortable with errors, retries and timeouts.
• Git, Linux and Docker. Your AI app has to run somewhere that isn't your laptop.
• A little SQL. A lot of AI work is getting the right data to the model.
• Reading other people's code. Most early AI work is wiring libraries and SDKs together. Being able to follow a repo and find out why it fails is worth more than a course.

What to skip for now: training models from scratch, deep math, building your own neural network. Learn them later if you love them. You don't need them to be productive this year.

If you're new to coding, budget two to three months. If you're already a developer or DevOps engineer, you're done. Go to stage 2.

Stage 2: Working with LLMs (tokens, temperature, structured output)

The most basic thing an AI engineer does is call a model. Here's a 1.5B parameter model running locally, called with a plain HTTP request:

$ curl -s localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "prompt": "In one sentence, what is Kubernetes?", "stream": false, "options": {"temperature": 0}}' | jq '{response, prompt_tokens: .prompt_eval_count, output_tokens: .eval_count, seconds: (.total_duration/1e9)}'
{
  "response": "Kubernetes is an open-source platform for automating the deployment, scaling, and management of containerized applications.",
  "prompt_tokens": 37,
  "output_tokens": 23,
  "seconds": 2.515247066
}

Look past the answer at the numbers. 37 tokens in, 23 out, two and a half seconds. Tokens are how models count text, roughly three quarters of a word each. You pay per token, you wait per token, and every model has a maximum number of tokens it can see at once (the context window). Tokens are your cost, your latency and your memory all at once.

Quick math to make it real: a support bot handling 10,000 questions a day, where each request (instructions plus retrieved documents) is about 2,000 tokens, uses 20 million tokens a day. That's 600 million a month. The price per million tokens suddenly becomes the most important number in your design. And bigger context windows aren't free: you pay for every token on every call, and models get worse at finding the important bit when it's buried in a huge prompt.

Temperature is randomness. We asked the same model to name a Kubernetes monitoring startup three times. At temperature 1.3 it said "KubeLens", then KubeWatchers. At temperature 0 it said "KubeLens" again, and keeps saying it. High temperature for creative work, low or zero for anything that touches production.

Structured output is the skill that quietly matters most. Give the model a messy log line, turn on JSON mode, and ask for fields:

$ curl -s localhost:11434/api/generate -d '{"model": "qwen2.5:1.5b", "prompt": "Extract service, error and exit_code as JSON from this log line: \"checkout-8b854d58f-jpsjt  Last State: Terminated  Reason: OOMKilled  Exit Code: 137\"", "stream": false, "options": {"temperature": 0}, "format": "json"}' | jq '{response, prompt_tokens: .prompt_eval_count, output_tokens: .eval_count, seconds: (.total_duration/1e9)}'
{
  "response": "{\n  \"service\": \"checkout\",\n  \"error\": \"OOMKilled\",\n  \"exit_code\": 137\n}",
  "prompt_tokens": 80,
  "output_tokens": 29,
  "seconds": 1.257905664
}

That isn't a chat answer. It's data your code can use. Once you can turn messy text into reliable structure, you can build real systems.

Also learn prompting (clear instructions, examples of good output, what to do when the model doesn't know) and treat prompts like code: version them and test them. And understand hallucination properly. The model isn't lying, it's predicting likely text. Your job is to give it the right information and let it say "I don't know".

Stage 3: RAG (retrieval augmented generation)

If you learn one thing from this roadmap, learn RAG. It's in almost every AI job description.

Models don't know your runbooks, docs or tickets, and if you ask anyway they'll invent something confident. RAG fixes that: search your own documents first, then hand the relevant bits to the model with the question.

Our version is tiny on purpose. Six on-call runbooks (OOMKilled, Pending pods, disk full, image pull errors, DNS, rollbacks), each turned into an embedding with nomic-embed-text, ranked by cosine similarity. The whole script is 27 lines, with no vector database and no framework:

$ python3 rag.py
question: my pod keeps dying with exit code 137, what should I do?
top matches:
  0.783  oomkilled.md
  0.624  pending.md
  0.568  dns.md
answer (grounded in oomkilled.md):
- Check kubectl describe pod for Last State OOMKilled to understand the cause.
- Compare pod's memory usage with kubectl top to identify potential leaks.
- Increase the memory limit or fix the underlying issue causing the OOMKilled condition.
- Do not restart the pod; address the root cause to prevent future OOMKilled events.
took 3.3s

It found the right runbook by meaning, even though the question never said "OOMKilled". The answer comes from that runbook, including the "don't just restart it" line.

Now count the bullets. We asked for three. It gave four. Small models don't always follow instructions, and that's exactly what evals and output checks are for.

What separates junior and senior work in RAG:

• Chunking. Cut a document in the wrong place and the answer is split in half and never retrieved. Most teams chunk by headings or paragraphs, with some overlap.
• Hybrid search. Meaning-based search is great for plain English and bad at exact strings like error codes and ticket numbers. Combine it with keyword search and merge the results.
• Reranking. Retrieve 20 candidates quickly, then use a more careful model to pick the best 3. Often the single biggest quality jump.
• Permissions. If your RAG can search every document, anyone who can ask a question can read everything. Filter by what the user can see, before retrieval.

Stage 4: Evals, the stage most people skip

With normal code you write tests: same input, same output. AI outputs change, so you build an evaluation set of real questions with expected answers, and score the system every time anything changes. We wrote six for the RAG app:

$ python3 eval.py
PASS container killed, exit code 137            -> oomkilled.md
PASS pod never gets scheduled                   -> pending.md
PASS df says 100% but du finds nothing          -> disk-full.md
FAIL ErrImagePull after deploy                  -> rollback.md
PASS cannot resolve external api from pod       -> dns.md
PASS undo the release that broke checkout       -> rollback.md
retrieval accuracy: 5/6

Five out of six. The image pull question retrieved the rollback runbook because it contained the word "deploy", and the rollback runbook is all about deploys. Without this test we'd have shipped it thinking it was perfect. Now there's a number to improve (better runbook text, a better embedding model, or hybrid search) and a test that proves it improved.

What to measure: hit rate for retrieval (did the right document appear in the top results), faithfulness for answers (does it only use what was retrieved), and relevance (did it answer the question). LLM-as-judge scales well, but judges prefer longer answers and answers that sound like themselves, so spot-check them with humans. Put the eval in CI so a prompt change that drops the score doesn't ship. In production, log questions and answers and track latency and cost per request. If you come from SRE, this is just observability for a new kind of service.

Stage 5: Agents, tools and MCP

An agent is a model that can call tools. You describe functions, the model decides when to use them, your code runs them, and the results go back to the model. We gave a model one read-only tool, get_pods, and asked a question:

$ python3 agent.py
user: Is anything broken in the shop namespace?
agent -> tool call: get_pods({"namespace": "shop"})
tool result:
NAME                       READY   STATUS             RESTARTS      AGE
...
checkout-8b854d58f-4f5zb   0/1     CrashLoopBackOff   3 (37s ago)   86s
...
agent: The `checkout` pod is experiencing issues, showing `CrashLoopBackOff`. This indicates that the pod is crashing and restarting repeatedly.

The important part: the model is not the security boundary. The tool runs kubectl as a service account with exactly one permission:

$ kubectl auth can-i list pods -n shop --as=system:serviceaccount:shop:agent-readonly
yes
$ kubectl auth can-i delete pods -n shop --as=system:serviceaccount:shop:agent-readonly
no
$ kubectl auth can-i delete deployments -A --as=system:serviceaccount:shop:agent-readonly
no

Even if someone tricks the model with a clever prompt, the worst it can do is read a pod list.

Learn MCP (Model Context Protocol), the standard way to plug tools into models so you write a tool once and any compatible assistant can use it. But know when not to build an agent. If the steps are always the same, write a normal workflow and call the model only where you need language. Agents are for when the next step depends on what you find, and they're slower, costlier and harder to test. If you build one: narrow tools with clear descriptions, a cap on steps, and a log of every tool call.

Stage 6: Shipping to production

This is where people from platform and DevOps backgrounds win.

• Serving. Hosted APIs are easier and usually smarter. Self-hosting gives you control and privacy, but now you're managing GPUs.
• GPU memory math. In our model list, qwen2.5:7b is 4.7 GB because it's quantized to roughly 4 bits per parameter. At full 16-bit precision it would be about 14 GB, before memory for the conversations it's serving.
• Serving engines. Tools like vLLM batch requests on one GPU, which is how you get throughput without a rack of hardware.
• Latency and cost. Stream tokens, cache repeated questions, and route easy jobs to a small model.
• Reliability. Timeouts, retries with backoff, a fallback model, and an SLO for the AI feature like any other service.

Stage 7: Security and guardrails

Treat every model input as untrusted, and every model output too. Prompt injection (in a user message or a retrieved document) can't be fully prevented with prompts, so limit what the model is allowed to do. Read the OWASP Top 10 for LLM applications once; it's short, and "excessive agency" is exactly the agent-with-too-much-power problem. Redact secrets and personal data before they reach a model you don't control. And require human approval for write actions like scaling, deleting or moving money.

A 6-month plan and 3 projects

1. Month 1: Python and APIs if you need them. Call a model from code and get JSON back reliably.
2. Months 2 and 3: A RAG app over documents you care about, plus an eval set of at least 30 questions. Make the score go up.
3. Month 4: An agent with two or three read-only tools and permission limits from day one.
4. Months 5 and 6: Ship it. Container, deployment, monitoring, cost tracking, a simple web page. Write up what broke.

Those three projects (a RAG app with real evals, a safe agent, a production deployment) beat ten certificates. What to skip: six months of math before building anything, chasing every framework that trends on a Tuesday, and demos that only work once.

Go deeper

Every command above comes from the [AI Engineer Lab](/labs/ai-engineer/), which has the RAG script, the eval, the agent and the RBAC manifest so you can run it all on a laptop. For the full walkthrough of all seven stages, including the code line by line, watch the [AI Engineer Roadmap 2026](/episodes/ai-engineer-roadmap/) episode.