AI Platform Engineering: MLOps and LLMOps on Kubernetes

This is for you if

• You're a platform engineer or SRE who has been asked to support data science and LLM teams
• You want to move into AI platform, ML platform or MLOps roles and need hands-on proof
• Your teams have scattered model API keys, no idea of LLM spend and fights over GPUs
• You need to answer 'which model is in production and who approved it?' with evidence

What's inside

• What an AI platform team owns, how the role differs from ML and AI engineers, and the ML lifecycle in platform terms
• MLflow 3 on Kubernetes: tracking, model registry with aliases, the Host-header security setting that trips upgrades, and lineage
• Data, features and training: feature stores and training/serving skew, Kubeflow Trainer, KubeRay and in-cluster training Jobs
• KServe v0.21 in Standard mode: InferenceService internals, runtimes, protocols, autoscaling and a registry-to-serving hand-off
• An LLM gateway with LiteLLM and Ollama: model aliases, fallbacks, per-team keys, rate limits, budgets and cost per request
• Evaluation pipelines and a prompt registry, with a working eval gate that decides whether a prompt version gets promoted
• GPU capacity with Kueue v1beta2: cohorts, borrowing, reclaim and priority preemption, plus chargeback and showback
• Model observability and governance: PSI drift checks, model cards, approvals and a ValidatingAdmissionPolicy for serving
• A golden-path Helm chart, a reference architecture, troubleshooting playbook, 24 interview Q&As, cheat sheets and a 30-day plan

After this guide you can

• Run MLflow on Kubernetes and promote models with aliases and approval tags instead of copying files
• Serve registered models with KServe and explain why the serving runtime is part of the model contract
• Put every LLM call behind a gateway with team keys, limits, budgets and visible cost
• Gate model and prompt changes with recorded evaluations
• Share scarce capacity fairly with Kueue queues, cohorts and preemption
• Design and defend a multi-team AI platform architecture in a design review or interview

Contents

1. What an AI platform team actually doesp. 5
2. Experiment tracking and the model registryp. 11
3. Data, features and training pipelinesp. 20
4. Packaging and serving modelsp. 28
5. The LLM gatewayp. 37
6. Evaluation pipelines and prompt managementp. 47
7. GPU capacity and multi-tenancyp. 53
8. Observability for modelsp. 60
9. Governance, golden paths and the reference architecturep. 66
10. Troubleshooting playbookp. 78
11. Production checklist and design decisionsp. 81
12. Interview questionsp. 84
13. Cheat sheet and 30-day planp. 93