Running LLMs on Kubernetes

This is for you if

• You run Kubernetes and your team now wants to self-host LLMs
• You need to size, justify and control a GPU bill with real numbers
• You're on call for model servers and want a playbook for slow, stuck or broken inference
• You're interviewing for platform, SRE or AI infrastructure roles

What's inside

• Prefill, decode, KV cache, PagedAttention and continuous batching explained in infrastructure terms
• GPU memory and KV cache formulas with worked examples, plus a calculator script that reads any model's config.json
• vLLM, SGLang, Ollama and llama.cpp compared, with TGI's maintenance-mode status and what to move to
• GPU scheduling end to end: GPU Operator, device plugin, labels and taints, time-slicing, MIG and DRA
• A production vLLM manifest explained field by field, plus a real GPU lab on GKE with costs and teardown
• Model storage and cold starts: OCI image volumes, mirrors, streaming loaders and a phase-by-phase cold start lab
• vLLM and DCGM metrics, the PromQL that matters and alert rules worth paging on
• KEDA autoscaling on in-flight requests, the scale-to-zero trap, and the Gateway API Inference Extension (InferencePool v1)
• Cost per million tokens, model provenance and network isolation, 24 interview Q&As, cheat sheets and a 30-day plan

After this guide you can

• Calculate whether a model fits a GPU and how many concurrent users it can serve
• Deploy vLLM or Ollama on Kubernetes with correct probes, storage and security settings
• Schedule and share GPUs with taints, selectors, time-slicing or MIG
• Autoscale inference on queue and in-flight metrics with KEDA and explain the cold start limits
• Monitor TTFT, inter-token latency, KV cache and GPU health with Prometheus
• Defend serving, routing and cost decisions in design reviews and interviews

Contents

1. How LLM inference really worksp. 4
2. The numbers: memory math, quantization and batchingp. 9
3. Serving engines: vLLM, Ollama, SGLang and the restp. 19
4. GPU scheduling on Kubernetesp. 27
5. Deploying vLLM with an OpenAI-compatible APIp. 35
6. Model storage and cold startsp. 43
7. Observability for LLM servingp. 49
8. Autoscaling and LLM-aware routingp. 57
9. Cost control and securityp. 67
10. Troubleshooting playbookp. 74
11. Production checklist and design decisionsp. 78
12. Interview questionsp. 80
13. Cheat sheet and 30-day planp. 89