71 real interview questions, answered

Sure. An SLI is the thing you measure. A service level indicator. For checkout, that might be the percentage of payment requests that succeed in under five hundred milliseconds.

The SLO is the target we set for that number. Say, ninety-nine point nine percent of requests succeed, measured over thirty days. That's internal. It's the line the engineering team holds itself to.

And the SLA is the promise we make to customers, usually with money attached. If we break it, we pay credits. So the SLA is always looser than the SLO. You want to find out you're in trouble long before the lawyers do.

Thirty days is forty-three thousand two hundred minutes. Point one percent of that is forty-three point two minutes. So roughly forty-three minutes of total failure a month. That's your error budget.

You spend it. That's the part people miss. If we've barely touched it, we can ship faster, run riskier experiments, do that migration. If we've burned through it, we slow down. Feature releases pause, and the team spends its time on reliability until the budget recovers.

It should be agreed before anything breaks. An error budget policy, signed off by engineering and product. Otherwise every freeze turns into an argument at the worst possible moment.

Toil is work that's manual, repetitive, can be automated, and doesn't make the system better over time. And it grows as the service grows. The classic one is a person restarting a stuck job every morning because it leaks memory.

Google's SRE book suggests keeping toil under fifty percent of an SRE's time. If you're above that, you're an operator, not an engineer. My move is to track it. Tickets tagged as toil, so you can show the trend and justify the automation work.

Honestly? That alert shouldn't wake anyone up. CPU at ninety-five percent is a cause, not a symptom. Customers don't feel CPU. They feel errors and slow pages.

So I'd page on symptoms. Error rate, latency, the SLO burning too fast. And keep CPU as a dashboard signal or a ticket for the morning. If high CPU actually hurts users, the symptom alert will catch it.

I've been that person. You get paged for a batch job that's supposed to use all the CPU. Twice. Then you stop trusting the pager.

One twenty-eight plus nine. Nine is SIGKILL. When a process dies from a signal, the exit code is one twenty-eight plus the signal number. And SIGKILL can't be caught, so the app never gets to log anything. It just disappears.

It's warming an eighty megabyte product cache, loads forty, and then... nothing. And the memory limit on this deployment is sixty-four megabytes. So version two can never start. It's not flaky. It's mathematically dead.

Roll back now, fix forward later. Undo the deployment to the previous revision.

Three version-one pods running, and the crashing pod is terminating. Let's run the test again.

Wait. Still forty out of forty failing.

Okay. Healthy pods but failing requests means the traffic isn't reaching them. So I stop looking at the pods and follow the request. Storefront calls the checkout service. Does the service have any endpoints?

None. The service is pointing at nothing. So the selector doesn't match the pods. Show me the service with its selector.

The selector is app checkout, version two. And we just rolled every pod back to version one.

Yep. Every pod is labelled version one. So the release pipeline didn't just change the deployment. It also patched the service to point at version two. When we rolled back the deployment, the service didn't roll back with it.

So the fix is to stop the service caring about version at all. It should select app checkout. Versioning belongs to the deployment, not the service.

Three endpoints. Now the test.

Okay. We're back.

...No one. I went straight into the terminal.

That's fair. Declare, post a status, then debug. And update every fifteen to thirty minutes even if the update is, still investigating.

Requests are what the scheduler reserves for you. Limits are the ceiling. Go over the memory limit, you get OOMKilled, like we just saw. Go over a CPU limit, you don't die, you get throttled. Which is sneaky, because the pod looks healthy and it's just slow.

Three of them. Guaranteed, when requests equal limits for everything. Burstable, when you set requests but they're not equal to limits. BestEffort, when you set nothing. When a node runs out of memory, BestEffort pods get evicted first, then Burstable. Our checkout pods should be Burstable right now.

Burstable. Memory request forty-eight, limit one sixty. If checkout was critical enough, I'd make it Guaranteed.

CoreDNS. Every service gets a DNS name, service dot namespace dot svc dot cluster dot local, and it resolves to the service's cluster IP. Then kube-proxy rules on the node send that traffic to one of the healthy pod endpoints. Let's look from inside the storefront pod.

Wait. Those aren't cluster IPs. That's the public internet. Dot shop is a real top-level domain, and this nslookup sent the name exactly as typed. Our app still works because the normal resolver tries the search domains first. Let's look at them.

Search domains, starting with shop dot svc dot cluster dot local. And ndots five. That ndots line is a famous performance trap. Any name with fewer than five dots gets tried against every search domain first. So calling an external API like api dot stripe dot com can fire off several DNS lookups before the real one.

A PodDisruptionBudget can block it. That's the point of a PDB. It says how many pods of an app are allowed to be down during voluntary disruptions, like drains and upgrades. If someone sets max unavailable to zero, the drain can never finish.

There it is. Cannot evict pod, as it would violate the pod's disruption budget. The drain is stuck, and the node's already cordoned, so nothing new lands on it either.

I don't delete the PDB on my own. It's there for a reason. I'd talk to the owning team, and the fix is a sane budget, like max unavailable one, with enough replicas that losing one is fine. Max unavailable zero isn't safety. It just means you can never patch.

Failed to pull image. It can't even resolve docker dot io, library, P-Y-H-T-O-N. Look at the name. It's a typo. Otherwise I'd check registry credentials, the image pull secret, and whether the tag exists. But first, always read the event. It usually tells you.

Because deleting a file only removes its name. The space isn't freed until the last process closes it. Classic cause: someone runs rm on a big log file, but the app is still writing to it. The file's gone from ls, but not from the disk. lsof plus L one lists open files with zero links.

There it is. A python process holding app dot log, marked deleted. Around four hundred and seventy megabytes.

That works, but it's checkout. I don't want a restart if I can avoid it. You can truncate the file through the process's file descriptor in proc. That frees the space instantly and the process keeps running.

Back to almost nothing used. Then the real fix: proper log rotation, with copytruncate or a signal so the app reopens its log, and an alert at eighty percent disk instead of a hundred.

The four golden signals. Latency, traffic, errors, and saturation.

Latency, but split by success and failure, because failed requests are often fast and they make your averages look great. Traffic, so you know the load. Errors, both explicit five hundreds and wrong answers. And saturation, how full the most constrained resource is.

Because averages hide pain. If ninety-nine requests take a hundred milliseconds and one takes ten seconds, the average looks fine. But that one user is having a terrible day. I'd look at the ninety-ninth percentile. And the users stuck in p99 are often your biggest customers, the ones with the most data.

Burn rate is how fast you're spending the error budget compared to the steady pace. A burn rate of one means you'll use exactly the whole budget in thirty days. A burn rate of fourteen point four means you'd use the whole month's budget in about two days.

The approach from Google's SRE workbook is multi-window alerts. Page if you're burning at fourteen point four times over the last hour, and still burning over the last five minutes. That second window stops the alert from paging after the problem's already gone.

Roughly, the error ratio is the rate of five hundreds over the rate of all requests. Then you compare it to fourteen point four times the allowed error, which is point one percent. And you do the same for the five-minute window and require both.

Metrics tell you something's wrong. Traces tell you where. Logs tell you why. In the incident we just did, the error rate told us something was wrong, following the request found the broken hop, and the pod logs showed the cache warm-up dying.

First, four nines is about four minutes and twenty seconds of downtime a month. That's less than one bad deploy. So it changes how you ship, not just how you build.

Second, the math of dependencies. Checkout calls the payment provider, inventory, and the database. If each one is ninety-nine point nine, and checkout needs all of them, you multiply. Point nine nine nine cubed is about ninety-nine point seven. You can't be more reliable than the things you hard-depend on.

You stop hard-depending on them. Inventory: if it's slow, take the order and reconcile after. The payment provider: two providers, with failover. And put the order on a durable queue, so if a downstream service is down, the order waits instead of failing.

Then redundancy. At least three availability zones, with enough capacity that losing one zone doesn't overload the other two. Database with synchronous replication across zones and automatic failover.

And every network call gets a timeout, retries with exponential backoff and jitter, and a circuit breaker. Without jitter, every client retries at the same moment and you get a thundering herd that keeps the service down.

Charging someone twice. If the request times out, you don't know if the charge happened. So every payment request carries an idempotency key. The provider sees the same key twice and returns the first result instead of charging again.

Small and slow. Canary to one percent, watch the SLO burn rate automatically, then ramp. Auto-rollback if the burn rate spikes. Feature flags so you can turn off code without a deploy. And everything we learned today: no version in service selectors, and test the image under real limits.

First, I don't fix anything. I measure. Every page for the last month, grouped by alert. Usually it's five alerts causing eighty percent of the noise.

Then for each noisy alert, one of three things. Delete it if nobody acts on it. Turn it into a ticket if it can wait until morning. Or fix the real problem if it's a genuine symptom. And I'd make a rule: every page needs a runbook link, or it doesn't page.

A healthy rotation. At least six to eight people so you're not on call every other week. Time off after a bad night. And a weekly on-call handoff where we look at what paged and why. Burnout is a reliability problem too. Tired people make the three a.m. mistakes.

We had a launch date fixed by marketing, and the load test showed the database falling over at about sixty percent of the expected traffic. I didn't just say no. I brought the graph, the error budget, and two options: launch to ten percent of users with a feature flag, or slip two days. They picked the flag. We fixed the query during the ramp, and nobody outside the team ever knew.

DevOps is a culture. Developers and operations owning software together, all the way to production. Platform engineering is one way to make that work at scale.

Instead of every team learning Kubernetes, Terraform, CI, secrets and monitoring from scratch, a platform team builds an internal product. Golden paths. A developer runs one command or fills one form, and gets a service with CI, deploys, dashboards and alerts already wired in. Done the right way by default.

The developers. And that changes everything. You do user research, you have a roadmap, you measure adoption. If nobody uses your platform, it doesn't matter how clever it is.

I'd make it optional but obviously better. Paved road, not a cage. If teams are routing around the platform, that's feedback. Something in it is slower or harder than doing it yourself. The exceptions are guardrails, like security policies, which apply to everyone.

The four DORA metrics are a good start. Deployment frequency, lead time for changes, change failure rate, and time to restore. Then platform-specific ones. Time for a new engineer to get their first change into production. Time to create a new service. And adoption, what percentage of services are on the golden path.

And I'd ask developers directly, every quarter. A short survey. Numbers tell you what, people tell you why.

Yes. The platform is production for every team on it. If the CI system or Argo is down, nobody can ship, and during an incident nobody can ship the fix. So I'd set SLOs on the things developers feel. Pipeline success rate, time from merge to running in production, and cluster API availability. And the platform team goes on call for them, like any other—

Not for everything. For the paths that block a production fix. If the deploy pipeline is down during an outage, that's a page. If a nightly report job fails, that's a ticket. Same rule as any service. Page on what hurts users, and here the users are engineers trying to fix production.

I'd look for the thing that's blocking the most people the most often. Usually it's not the shiny request. It's something boring, like environments taking two days to create, or every team writing its own CI pipeline. Fix the paved road for the common case first, and give the rare cases a documented way out. And say no out loud, with a reason, so people aren't waiting on a ticket that's never going to happen.

Kubectl reads the YAML, and talks to the API server over HTTPS. The API server authenticates you, checks RBAC, runs admission controllers, validates the object, and stores it in etcd.

Okay. First it fetches the OpenAPI schema, so kubectl knows what a deployment looks like. Then a GET for the deployment called hello. Four oh four, it doesn't exist yet. So it sends a POST, and gets two oh one, created. That's client-side apply. Check if it exists, then create or patch.

Right. At that point it's just a record in etcd. Then the controllers take over. The deployment controller sees a new deployment and creates a ReplicaSet. The ReplicaSet controller creates pods. The scheduler sees pods with no node and picks one. Then the kubelet on that node—

No. Nothing calls the controllers. They watch. Each controller has a long-running watch on the API server for the objects it cares about, and keeps a local cache. When something changes, it compares desired state with actual state and acts on the difference. That's the reconcile loop. And it's level-based, not event-based, so if a controller misses an event, the next resync still catches the difference.

Two phases. Filtering, then scoring. Filtering removes nodes that can't run the pod. Not enough CPU or memory requested, a taint the pod doesn't tolerate, a node selector or affinity that doesn't match, a volume in a different zone. Then scoring ranks what's left, like spreading pods or packing them, depending on config. Highest score wins, and it writes the node name onto the pod. That's called binding.

Requested. The scheduler only looks at requests, never at real usage. A node can be at five percent real CPU and still be full, if the requests add up. That's why bad requests waste so much money.

Two different things can happen. If a container goes over its own memory limit, the kernel kills it. That's OOMKilled, exit code one three seven. But if the whole node runs low, the kubelet starts evicting pods before the kernel has to step in. It picks by quality of service. Best effort pods first, the ones with no requests at all. Then burstable pods using more than they asked for. Guaranteed pods last.

Requests equal to limits, for both CPU and memory, on every container in the pod. That's why critical system pods often look like that.

Topology spread constraints. Topology key is the zone label, max skew one, and when unsatisfiable, do not schedule. Max skew one means the difference between the busiest zone and the emptiest zone can't be more than one pod.

Two, two and two. Perfect.

Do not schedule is strict. If a zone is down, new pods stay pending rather than piling into the surviving zones. That protects your spread, but it can block a rollout. Schedule anyway is a preference. For most stateless services I'd actually use schedule anyway, plus enough replicas, so losing a zone degrades you but doesn't block you.

By default, nothing. The checkout pod sits in pending until something frees up. Unless you've set up priority. Give checkout a high priority class and the batch jobs a low one, and the scheduler is allowed to preempt. It evicts lower-priority pods to make room.

Pending for a moment, with no node. And there it is. The scheduler preempted one of the batch pods. That's the event, preempted by the checkout pod, on the first worker.

Two big ones. If everyone marks their service as critical, priority means nothing. So the platform team owns the priority classes, and teams can't create their own. And preemption ignores some of your safety. It tries to respect pod disruption budgets, but it won't wait forever. So a low-priority service can lose pods suddenly. That's acceptable for batch, not for anything stateful.

A CRD, a custom resource definition, teaches the API server a new kind of object. Like an Argo CD application, or a Postgres cluster. Just the data. An operator is a controller that watches those objects and does the work. The same reconcile loop we talked about, but for your own kind of thing.

Kubernetes has one rule. Every pod gets its own IP, and every pod can reach every other pod without NAT. How that happens is the CNI plugin's job. On EKS, the default is the VPC CNI, which gives pods real IP addresses from the VPC subnet. So to the network, a pod looks like any other machine.

You run out. Every pod uses an IP from the subnet, and each instance type can only hold a limited number of network interfaces and addresses. Small subnets fill up fast. The fixes are bigger subnets, prefix delegation so each interface gets blocks of addresses, or a secondary CIDR range just for pods.

Nowhere. That's the trick. No interface has that IP. Kube-proxy on every node watches services and their endpoints, and writes rules, iptables or IPVS, or eBPF if you use something like Cilium. When a packet goes to the ClusterIP, the node rewrites the destination to one of the healthy pod IPs.

Usually nothing. The rules match the service port and protocol, not ICMP. People see the ping fail and think the service is broken. Test it with curl on the actual port instead.

Because network policies are just objects. Something has to enforce them, and that's the CNI. If your CNI doesn't support network policies, they're ignored silently. On EKS you have to turn on network policy support in the VPC CNI, or use something like Calico or Cilium.

DNS. People deny all egress, then forget to allow traffic to CoreDNS on port fifty-three, UDP and TCP. Suddenly nothing resolves and every service looks down.

Usually through the AWS Load Balancer Controller. An ingress or a gateway creates an Application Load Balancer. In IP mode, the load balancer sends traffic straight to pod IPs, skipping the extra hop through the node. And for TLS, certificates from ACM on the load balancer, or cert-manager inside the cluster.

HPA changes the number of pods. More replicas when CPU or a custom metric goes up. VPA changes the size of pods, their requests. Those two scale the workload. The cluster autoscaler and Karpenter scale the nodes. When pods can't be scheduled, they add machines, and when machines are empty, they remove them.

The cluster autoscaler works through node groups. On AWS that's auto scaling groups, each with a fixed instance type. It decides which group to grow, and the group launches a node. Karpenter skips the node groups. It looks at the pending pods, works out what they need, and launches the right instance directly from a wide range of types. So it's usually faster, it can pick a bigger or cheaper machine for the actual workload, and you're not managing dozens of node groups.

The NodePool says what Karpenter is allowed to launch. AMD64, on-demand only, from the C, M or R families, with fewer than nine CPUs. The node class points at the EC2 settings, the AMI, subnets and security groups, found by tag. Then limits, CPU eight, so it can never buy more than eight CPUs in total. And the disruption block, consolidate when empty or underutilized, after one minute.

Because without it, a bug that creates ten thousand pods becomes ten thousand dollars. The limit is your blast radius for cost. Every NodePool should have one.

All five pending. The existing nodes can't fit them.

And Karpenter has already created a NodeClaim. That's Karpenter's record of a machine it's launching.

Found provisionable pods. Computed a new NodeClaim. Created it, launched it, then registered and initialized. Thirty seconds from pending pods to a ready node. And it picked one c6a two x large for all five pods, instead of five small machines. That's bin-packing.

That's consolidation. Karpenter noticed the node was empty, tainted it so nothing new lands there, and about thirty seconds later the node and the NodeClaim were gone. The instance is terminated, so you stop paying for it.

A few things. First, pod disruption budgets on every service, so Karpenter can only take down a limited number at once. Second, disruption budgets on the NodePool itself, like at most ten percent of nodes at a time, or no disruption during business hours. Third, for something that truly can't be interrupted, like a long batch job, the do-not-disrupt annotation on the pod.

Then you've turned off consolidation and you're paying for empty machines. It's an exception, not a default.

Yes. It runs a scheduling simulation that understands spread constraints, affinity and taints. If your pods must spread across three zones, it can launch nodes in three zones. That's one of its advantages over node groups that are tied to one zone. But it can only use the zones your subnets allow, so if the node class only finds subnets in two zones, a three-zone hard spread will leave pods pending.

To keep nodes fresh. After thirty days, Karpenter replaces the node, so you pick up patched AMIs and you never have a server that's been running for two years that nobody dares to touch. It forces your workloads to survive node replacement all the time, which is exactly what you want to find out before an incident, not during one.

AWS gives a two-minute warning. Karpenter listens for those through an SQS queue fed by EventBridge. That's the interruption queue setting in the Helm chart. When the warning comes, it cordons the node, launches a replacement, and drains the pods, inside those two minutes.

And I'd only put workloads on spot that can handle it. Stateless, multiple replicas, spread across zones and across many instance types. The more types you allow, the less likely they're all reclaimed at once.

Not on nodes it manages. If Karpenter is running on a node it decides to remove, you can end up with nothing left to launch capacity. That's why this cluster has a small managed node group, the two original nodes, just for Karpenter and other critical system pods. Or you run it on Fargate. Everything else can go on Karpenter nodes.

The pod events first. Then the Karpenter logs, because it will usually say why. The common causes are: the NodePool requirements don't match what the pod asks for, like the pod wants ARM and the pool only allows AMD64. The NodePool limit is already reached. The pod has a taint toleration or node selector no pool satisfies. Or on the AWS side, there's no capacity for those instance types, or the account hit its vCPU quota.

Everyone hits it once.

Git is the source of truth for what should be running, and a controller in the cluster continuously makes the cluster match Git. You get history, review and rollback for free, and nobody needs admin credentials to the cluster to deploy.

Push is your CI pipeline running kubectl apply against the cluster. That means CI needs cluster credentials, and if someone changes the cluster by hand, nothing notices. Pull is an agent inside the cluster, like Argo CD or Flux, pulling from Git. The credentials stay inside, and it keeps checking for drift.

Argo noticed. The app is out of sync, and the diff shows exactly which resource. The deployment. The service is still fine.

Because self-heal is off. Auto-sync only applies new commits from Git. Self-heal is what reverts manual changes. Turn it on and Argo puts it back.

In production, I'd say yes, with one exception. During an incident, someone might scale up by hand to survive, and self-heal would fight them. So you either make the emergency change through Git, which can be fast if your pipeline is fast, or you have a documented way to pause sync for that app. What you don't want is a cluster that quietly drifts away from Git for months.

They fight. The HPA scales to ten, Argo sees ten instead of three, and puts it back. So you remove replicas from the manifest when an HPA owns it, or tell Argo to ignore that field.

Never as plain text in Git. Two common patterns. Encrypt them, like Sealed Secrets or SOPS, so only the cluster can decrypt. Or better at scale, keep them in a secret manager like AWS Secrets Manager or Vault, and use the External Secrets Operator to sync them in. Git only holds a reference to the secret, not the value.

I prefer one folder per environment, not one branch per environment. Branches drift and merges get scary. The pipeline builds the image once, then promotes the same image tag by opening a pull request that bumps the version in the staging folder, then production. Production needs an approval. And the image is never rebuilt between environments, so what you tested is what you ship.

Git revert. The controller applies the old version. And for risky changes, I'd add progressive delivery, like Argo Rollouts, so a canary takes a small slice of traffic first and rolls back automatically if error rates go up.

State is Terraform's memory. It maps every resource in your code to the real thing it created, with its ID and attributes. Terraform compares three things: your code, the state, and what's really there. People get nervous because if state is lost or corrupted, Terraform forgets it owns things, and might try to create them again.

Remote. An S3 bucket with versioning and encryption, with locking so two people can't apply at the same time. Newer Terraform versions can lock with S3 itself, older setups use a DynamoDB table. Never in Git, because it can contain secrets.

Two resources added, and both are in state now.

Terraform refreshed, saw the quota is gone, and plans to create it again. That's drift detection. The plan is the most important review step in Terraform. You read it before every apply.

Run plan on a schedule in CI, every night, with detailed exit codes. Exit code two means there are changes, so you alert the team. And the long-term fix is cultural. Remove write access in the console, so changes go through code.

Split it. By layer and by blast radius. Networking in one state, clusters in another, each team's app infrastructure in its own. Things that change together live together. Smaller states mean faster plans, and a mistake in one can't destroy everything.

Outputs. Read them with a remote state data source, or better, look things up by tag or from a parameter store, so the layers aren't tightly coupled. Then modules for anything repeated, versioned, so a change to the module doesn't hit every team on the same day.

Import. Write the resource block, then an import block with the bucket's ID, and run plan. The goal is a plan that shows no changes. If it wants to change things, adjust the code until it matches reality, then apply. Never just apply an import plan that modifies the resource you don't fully understand yet.

With OIDC. GitHub Actions or GitLab gets a short-lived token, and AWS trusts it for one specific role, limited to one repository and branch. No access keys stored anywhere. Long-lived keys in CI secrets are one of the most common ways companies get breached.

Layers again. Pin dependencies with lock files, so builds are repeatable. Scan images for known vulnerabilities in the pipeline, and fail on critical ones with a fix available. Generate an SBOM, a software bill of materials, for every image, so when the next big vulnerability lands, you can answer which services are affected in minutes, not weeks.

Sign it. The pipeline signs every image with something like cosign, using keyless signing tied to the pipeline's identity. Then an admission policy in the cluster only allows signed images from our registry. If someone pushes an image by hand, it can't run. And deploy by digest, not just by tag, because a tag can be moved.

Measure first. Which steps take the time. Usually it's the same few things. No caching of dependencies or Docker layers. Running every test for every change, instead of only what changed. Tests running one after another, instead of in parallel. And slow, shared runners. Fixing caching alone often cuts it in half. And I'd track CI duration as a platform metric, because slow feedback quietly kills productivity.

Layers. Each team gets its own namespace. Resource quotas, so one team can't take all the CPU. Limit ranges, so pods get default requests. RBAC, so a team can only touch its own namespace. Network policies, so traffic is denied by default between namespaces. And admission policies, for rules everyone has to follow.

Two out of three ready. The third pod was rejected at admission. Exceeded quota. It's not pending, it was never created. That's an important difference when you debug. You won't find it with get pods. You find it in the ReplicaSet events.

Can create deployments in team payments, yes. In the guestbook namespace, no. And they can't create resource quotas, even in their own namespace. So they can't raise their own limits. That's what you want.

Latest isn't a version. Two nodes can pull two different images under the same tag, and you can't roll back to latest. So you block it with an admission policy. Kyverno or Gatekeeper. Here, a Kyverno policy.

Denied at the door, with a message telling the developer exactly what to fix. And with a pinned version, it's allowed.

I went straight to enforce. Every new policy should start in audit mode. Report violations, don't block. Then share the report with teams, give them time to fix, and only then switch to enforce. And test policies in CI against real manifests, so you know the blast radius before it's live.

When the isolation needs are hard. Regulated data, like payments under PCI. Very different Kubernetes versions or upgrade schedules. Untrusted code from customers. Or one team so large it's noisy for everyone. Namespaces are soft isolation. Separate clusters are hard isolation, but each one costs you upgrades, monitoring and money.

It depends on who's going to operate it. If you're on AWS and you don't have database expertise, a managed service like RDS or Aurora is usually the right answer. Backups, failover and patching are someone else's job.

But it can be done well now, with a good operator. CloudNativePG, for example. You get consistency with everything else on the platform, it can be cheaper, and it works the same on any cloud. The cost is you own the operations, including the scary night when it fails.

One primary, two replicas. Each with its own persistent volume.

They shouldn't pick a pod. The operator creates services. R W always points at the primary, for writes. R O points at replicas, for read-only traffic. And R is any instance. When the primary changes, the service changes with it, so the app config never does.

Instance two is the primary now. The operator promoted a replica. And instance one came back as a replica.

Reading from the new primary. Keyboard and monitor. Both orders survived.

No. By default, replication is asynchronous. The primary confirms a write before the replicas have it. If the primary dies at the wrong moment, the last few transactions can be lost. You can turn on synchronous replication, so a write waits for at least one replica. Then you lose nothing, but every write is a bit slower, and if the replicas are down, writes can block.

RPO, recovery point objective, is how much data you can afford to lose, measured in time. RTO, recovery time objective, is how long you can be down. Synchronous replication gets your RPO close to zero for a single failure. Fast automatic failover gets your RTO down to seconds. And the business picks the targets, because each step costs money.

Because replication copies mistakes too. Someone runs delete without a where clause, and it's deleted on every replica within a second. A backup is a copy from the past. So you need base backups plus the write-ahead log archived to object storage, like S3. That gives you point-in-time recovery. Restore to two forty-nine, one minute before the bad delete.

That's the right question. A backup you've never restored is just a hope. I'd automate a restore every week into a separate cluster, check the data, and alert if it fails.

Carefully, and rehearsed. The safest pattern is a new cluster on the new version, kept in sync with logical replication, then a short cut-over. Point the read-write service at the new primary, and keep the old cluster around for a quick rollback. An in-place upgrade can be faster, but it's harder to undo. Either way, test it against a restored copy of production first, because extensions and query plans can change between versions.

Because the volume is in zone A. An EBS volume only lives in one zone, so the pod can only attach it in that zone. That's why you don't rely on moving a database volume between zones. You run replicas in each zone and fail over to one that already has its data. And the storage class should use wait for first consumer, so volumes get created in the zone where the pod is actually scheduled.

That's four thousand connections. Each Postgres connection is a process with its own memory. Most databases are tuned for a few hundred. You need a connection pooler in between, like PgBouncer, in transaction mode, so thousands of app connections share a small number of real database connections. CloudNativePG can run one for you, called a pooler.

Never in one step. Expand and contract. First, add the new column. Then deploy code that writes to both columns. Backfill the old rows in small batches. Then switch reads to the new column. And only when nothing uses the old column anymore, drop it. Each step is safe to roll back. A single rename would break every running pod that still uses the old name.

Progressing usually means the new pods never became ready. So I'd look at the rollout status, then the new ReplicaSet's pods. Three common answers. The pods are pending, so it's capacity or scheduling, and the events will say why. The pods are running but not ready, so the readiness probe is failing, often a wrong port or path. Or they're crash looping, so it's the logs from the previous container.

Then the app isn't listening where the probe is looking. Either the port changed in the new version, or the app listens on localhost only, so nothing outside the container can reach it. I'd exec in and check what it's actually listening on. And the fix goes in Git, not in the cluster, or Argo will just revert it.

Something shared changed. The registry, or the path to it. I'd look at one pod's events for the exact error. If it's unauthorized, the pull credentials expired or a token rotated. On EKS pulling from ECR, it's the node role's permissions. If it's a timeout, it's network. A NAT gateway, a VPC endpoint, or a firewall rule. And if it's too many requests, it's Docker Hub rate limits, which is why a platform should run a pull-through cache.

Then short term, the pods already running are fine, so I'd pause non-urgent deploys. Then point images at a mirror or pull-through cache, like ECR pull-through cache for Docker Hub. And afterwards, a policy that images must come from our own registry, so this can't happen again.

Something is hammering it. The usual suspect is a controller or an operator listing everything, over and over, without a cache. Or a script in a loop. I'd look at API server metrics by user agent and by resource, to find who's making the most requests. And on EKS, the control plane logs and audit logs show it. Then there's API priority and fairness, which can stop one noisy client from starving everyone else.

Then it's usually too many objects or very large ones. Thousands of old ReplicaSets, completed jobs that never get cleaned up, or huge config maps. Set revision history limits, set a time to live on finished jobs, and keep big data out of etcd.

I'd start with questions, not boxes. What hurts today? How long does it take to ship a change? How many incidents come from deploys? Which teams are happy and which are stuck? The platform should solve the top three problems, not everything.

Okay. Then the layers. At the bottom, infrastructure. Terraform modules, one account per environment, EKS clusters per region and per environment, with Karpenter for nodes.

On top, the delivery layer. One standard CI template, image build and scan, and Argo CD for deploys, with promotion through pull requests.

Then shared services built in. Logging, metrics and tracing with OpenTelemetry. Secrets through External Secrets. Policy with Kyverno. Certificates and DNS automated.

And at the top, the developer portal. Something like Backstage, with a service catalog so every service has an owner, and templates. A team clicks new service, and gets a repo, a pipeline, a namespace, dashboards and alerts in about ten minutes.

I wouldn't. Good catch. The portal is the last thing, not the first. It's just a front door. I'd start with the golden path itself. One working template for the most common service type, used by two friendly teams. Prove it saves time. The portal goes on top once there's something worth putting in it.

As few as you can get away with. Production and non-production separated, at minimum. Per region if you serve multiple regions. Split further only for hard isolation, like the PCI case. Every extra cluster is another thing to upgrade three times a year.

That depends on what the business paid for, and I'd make them choose explicitly. The cheap option is backup and restore. Infrastructure as code and data backups copied to a second region, so we can rebuild in hours. The middle option is a pilot light. A small cluster already running in the second region, database replicas there, scale it up when needed. The expensive one is active-active, both regions serving traffic, which is hard mainly because of the data.

Everything to rebuild has to live outside the failed region. Terraform state, container images, Git, secrets. If your image registry only exists in the region that's down, you can't start anything anywhere. I'd replicate the registry and the secrets, and practice the failover once a quarter. A disaster plan you've never run is fiction.

Read the changelog for removed APIs, and scan the manifests with a tool like Pluto before you start. Upgrade non-production first and let it soak. Upgrade the control plane, then the add-ons, then the nodes. With Karpenter, you bump the AMI version in the node class, Karpenter sees the nodes have drifted, and replaces them gradually, respecting the disruption budgets and the pod disruption budgets. And never be more than one version behind, because skipping versions isn't allowed and the backlog only grows.

Make it visible first. Cost per team and per namespace, using labels and a tool like Kubecost or OpenCost. Then the usual wins. Right-size requests, because most are set way too high. Consolidation with Karpenter. Spot for stateless work. Savings plans for the steady baseline. And turn off non-production at night and on weekends. When teams see their own bill, behaviour changes fast.

Liveness failing restarts the container. Readiness failing removes it from the service. Never point liveness at a database check, or a database blip restarts everything.

Requests are for scheduling and fair share. A CPU limit throttles, even when the node is idle. Many teams set CPU requests, memory limits, and no CPU limit.

Usually a finalizer that never finished, or a node that's gone so the kubelet can't confirm. Check finalizers before force deleting.

Gateway API is the newer standard. Split roles, so the platform owns the gateway and teams own their routes, and it supports more than HTTP.

Not by default. If you need mutual TLS everywhere, fine-grained traffic shifting or deep traffic telemetry, maybe. Otherwise it's a lot of moving parts for problems you don't have yet.

EKS Pod Identity or IRSA. A role per service account. Never node-wide credentials. That's how Karpenter itself gets its permissions in this cluster.

Helm for packaging things you install from others. Kustomize for patching your own manifests per environment. Plenty of teams use both.

Documentation that's actually up to date. A golden path nobody can find doesn't exist.

We moved forty services from an old deploy system to Argo CD. I planned it team by team, and the first five went perfectly. Then we hit the team with the most traffic, and their deploys suddenly took twice as long. Turned out their old system skipped health checks, and Argo was correctly waiting for readiness probes that took ninety seconds.

The mistake was mine. I'd tested with easy teams. So we paused, fixed their probes together, and I changed the plan. For each team, a dry run in staging first, plus a checklist of what's different. The rest of the migration took three weeks longer than planned, but nobody had an outage.

Pick one hard team early, not last. The easy teams make you feel fast, but the hard team is where you learn what the migration really involves.

I'd sit with them and watch them deploy, their way and our way. Sometimes they're right, and that's free product research. If their way really is faster, I want to know why and copy it. If ours is better for the company, like security or cost, I explain the trade-off honestly. And I'd never escalate first. Forcing people onto a platform makes enemies. Making it better makes fans.