Kubernetes Day-2 Operations: How to Run an Operational Readiness Review

Getting workloads into Kubernetes is only the beginning. The harder question is whether the environment can stay reliable while people patch it, scale it, troubleshoot it, rotate access, drain nodes, recover from failures, and keep applications changing underneath it.

That is the Day-2 problem.

A cluster can look healthy on a dashboard and still be operationally fragile. You can have green Pods with no usable recovery procedure. You can have autoscaling enabled while bad resource requests make scaling decisions unreliable. You can have PodDisruptionBudgets and still discover that node-pressure eviction behaves differently from planned maintenance.

The command is usually the easy part. The decision is knowing what evidence you need before you make the change.

This article gives you a repeatable operational readiness review. It is not an official Kubernetes framework. It is a practical operating model that combines documented Kubernetes behavior with clearly labeled professional judgment about how to organize the review.

I review Kubernetes Day-2 readiness across six domains:

Kubernetes documents observability around metrics, logs, and traces because each signal answers a different part of the health question. The review should connect those signals to operational decisions instead of treating monitoring as a dashboard project.

Start read-only.

A useful Day-2 review should not begin with a remediation sprint. Begin by proving what is true about the cluster you have now. That reduces the chance that the review itself becomes the cause of the outage.

A small evidence collection pass might include:

If the cluster exposes the Kubernetes resource metrics pipeline, add:

kubectl top depends on Metrics Server or another implementation of the Metrics API, so failure of that command is not automatically proof that the cluster is unhealthy. It is evidence that you need to understand how resource metrics are provided in that environment.

Do not confuse Kubernetes Events with durable incident history either. Events are useful, but Kubernetes treats them as best-effort, supplemental data with limited retention. If your operating model depends on reconstructing what happened last week, your logging, monitoring, or audit pipeline needs to carry that history.

The goal of the first pass is simple: establish what you can observe before you decide what to change.

The first question is not "Do we have monitoring?" It is "Can an operator move from symptom to evidence?"

For each production-critical workload, I want to know whether the team can answer:

•  Is the workload receiving traffic?

•  Is it ready to receive traffic?

•  Is it restarting?

•  Is it slow because of application behavior, resource pressure, dependency failure, or infrastructure?

•  What changed immediately before the symptom appeared?

•  Can we connect cluster symptoms to workload symptoms without switching between disconnected sources and guessing?

Kubernetes defines readiness, liveness, and startup probes for different purposes. Readiness controls whether a Pod should receive normal Service traffic. Liveness can cause a container restart. Startup probes protect slow-starting containers from being judged by liveness checks too early.

That distinction matters operationally. A probe that answers the wrong question can create the failure it was supposed to detect.

•  Are critical workloads using probes that represent real application health rather than shallow process checks?

•  Are readiness and liveness testing different conditions when they should be?

•  Can operators see container restarts, failed scheduling, image pull errors, OOM behavior, and node pressure?

•  Are metrics, logs, and traces retained long enough for the incidents the team is expected to investigate?

•  Are alerts tied to an operator action, or are they simply reporting that something is red?

If the team cannot distinguish application failure from cluster failure, avoid broad remediation. Improve the evidence path first. Changing capacity, probes, networking, and controllers at the same time will make the next symptom harder to explain.

Day-2 work means touching a running system. Upgrades, node maintenance, add-on changes, policy changes, certificate rotation, autoscaler tuning, and workload migrations all introduce risk.

The safe pattern is:

1.  define the expected result,

2.  prove the preconditions,

3.  make the smallest reasonable change,

4.  validate the result,

5.  stop or roll back if the evidence moves the wrong way.

For node maintenance, Kubernetes provides kubectl drain to evict workloads safely rather than simply powering off a node. A successful drain can respect PodDisruptionBudgets and graceful termination behavior. That makes disruption policy part of the precheck, not something you discover halfway through maintenance.

•  Is there a documented maintenance path for worker nodes?

•  Does the team know which workloads have disruption budgets and which do not?

•  Do maintenance procedures define a success check before the next node is touched?

•  Is there a stop condition when workloads cannot reschedule or recover?

•  Are rollback steps specific enough that another operator could execute them?

•  Are changes sequenced so you can identify which change caused a regression?

A PodDisruptionBudget is not a universal availability guarantee.

It constrains voluntary disruptions that use the eviction mechanisms Kubernetes can coordinate. Node-pressure eviction is different. The kubelet can evict Pods under pressure and does not respect a configured PodDisruptionBudget in that path.

That means "we have PDBs" is not enough evidence for resilience. You also need to understand node pressure, resource sizing, replica placement, and recovery behavior.

Capacity reviews fail when teams look only at current utilization.

Kubernetes scheduling is strongly influenced by resource requests. The scheduler uses requests when deciding where a Pod can run. Node autoscaling also reasons from scheduling constraints and Pod resource requests rather than simply looking at actual usage after the Pod starts.

So I review three different things together:

•  requested capacity: what workloads ask Kubernetes to reserve,

•  observed usage: what workloads actually consume,

•  placement constraints: what prevents otherwise available capacity from being usable.

A cluster can have apparently idle CPU and still have Pending Pods because requests, affinity rules, taints, topology requirements, storage constraints, or extended resources make the remaining capacity unusable.

•  Which Pods are Pending, and why?

•  Are requests missing, obviously stale, or far out of line with observed usage?

•  Are critical workloads concentrated on too few nodes or zones?

•  Does node autoscaling have the information it needs to make sensible provisioning decisions?

•  Are DaemonSets, storage requirements, GPUs, architecture constraints, taints, or affinity reducing usable headroom?

•  Are teams treating kubectl top as one signal rather than the entire capacity model?

Kubernetes documents that requests influence scheduling. It also documents that node autoscaling decisions depend on those requests. Requests that are too low can produce poor provisioning outcomes; requests that are too high can prevent useful consolidation.

That does not mean every workload needs perfect sizing before the cluster is safe. It means request quality is operational data, not a deployment-time detail you can forget forever.

Resilience is not "we have three replicas."

The review should ask what happens when the environment loses something the application depends on: a Pod, a node, capacity in a failure domain, network reachability, a dependency, or an operator-maintained component.

Start with the workload behavior you can actually prove.

•  Do critical workloads have enough replicas for their availability objective?

•  Are replicas spread in a way that avoids obvious single-node or single-zone concentration?

•  Do PodDisruptionBudgets match the maintenance behavior the team expects?

•  Do readiness probes remove unhealthy instances from normal traffic quickly enough?

•  Are liveness probes restarting containers only when restart is genuinely the right recovery action?

•  What happens when a node is under memory, disk, or PID pressure?

•  Can the workload reschedule if one node disappears?

Planned disruption and pressure-driven eviction are not the same operational event.

For a planned node drain, Kubernetes can coordinate graceful eviction and respect disruption budgets. Under node pressure, the kubelet protects node stability and can terminate Pods without respecting those budgets.

That difference should change how you validate resilience. Test maintenance safety through the maintenance path, and review node-pressure exposure through resource behavior and node health. Do not use evidence from one path as proof for the other.

A Kubernetes cluster has a lifecycle whether or not the team has a lifecycle process.

The Kubernetes project maintains the most recent three minor release branches, so support status is an ongoing operational check. Exact maintained versions and patch releases are time-sensitive and should be verified when the review is performed.

I would include version currency in the same Day-2 backlog as reliability work because unsupported or badly lagging versions make every future maintenance decision harder.

•  Is the control plane on a currently supported Kubernetes minor for the deployment model in use?

•  Are worker-node and kubectl versions within supported compatibility expectations?

•  Is there a predictable patch and upgrade cadence?

•  Are deprecated APIs and add-on compatibility checked before upgrades?

•  Is the team clear about which lifecycle responsibilities belong to the managed provider and which remain with the platform team?

Security belongs in the same recurring operating discipline.

Kubernetes publishes a Security Checklist as a baseline and explicitly warns that a checklist is not sufficient on its own. That is the right posture for a Day-2 review too. The goal is not to tick every box once. The goal is to identify controls that can drift and define how they are checked again.

Review access, network exposure, workload security posture, secrets handling, certificate lifecycle, and policy enforcement according to the environment's risk model.

Auditing is especially useful because Kubernetes audit records can provide a chronological security-relevant record of actions by users, applications, and the control plane. Availability and retention still depend on how the cluster or managed service is configured, so "Kubernetes supports auditing" is not evidence that your cluster has usable audit history.

Backups are not the outcome. Recovery is the outcome.

For self-managed clusters that use etcd directly, Kubernetes documentation makes the importance of periodic etcd backup explicit because etcd stores Kubernetes object state. It also documents snapshot verification and restoration behavior.

Managed Kubernetes changes the mechanics, but not the operational question. You still need to know what the provider protects, what you must protect separately, what restore options exist, and which application data lives outside the Kubernetes control-plane state.

•  Who owns control-plane or cluster-state backup?

•  What does the provider recover automatically, and what requires operator action?

•  Are application data backups separate from cluster-state backups where appropriate?

•  Is the restore procedure documented and current?

•  Are credentials, certificates, encryption keys, storage snapshots, DNS, load balancers, or external secrets dependencies included in recovery planning?

•  Has the recovery path been validated safely in a non-production environment or through a provider-supported recovery exercise?

If nobody can explain the recovery path for a critical environment, treat that as a readiness gap before performing high-risk lifecycle work. A rollback plan that depends on a restore process nobody has validated is not a strong rollback plan.

An at-a-glance recap of the six-domain model and the operator outcome the review is designed to produce.

This is where the six domains become useful.

Do not finish the review with a long list of observations. Every meaningful finding should become one of three types of work. These categories are professional judgment for organizing the backlog, not Kubernetes-defined severity levels.

Use this when a credible failure path exists, and the team lacks a safe containment, rollback, or recovery path.

Examples:

•  a critical workload has no workable recovery path,

•  node maintenance cannot complete without violating availability expectations,

•  operators cannot tell whether a production symptom is application or cluster failure,

•  a material lifecycle change is required, but rollback assumptions are unverified.

Use this when the environment is operating but the evidence shows recurring operational exposure.

Examples:

•  resource requests consistently undermine scheduling or scaling decisions,

•  critical workloads have weak disruption coverage,

•  audit data exists but is not retained or accessible for investigation,

•  the cluster is drifting toward the end of its supported lifecycle window.

Use this after the safety posture is credible.

Examples:

•  automate recurring evidence collection,

•  improve dashboard-to-runbook links,

•  reduce noisy alerts,

•  standardize readiness review output,

•  teach application teams how requests and probes affect platform behavior.

Avoid arbitrary numeric risk scoring unless your organization already has a defensible model. The evidence quality, failure consequence, blast radius, reversibility, and recovery confidence are usually more useful than a made-up score.

Use this as the working review artifact.

□  Critical workload health can be traced through metrics, logs, and relevant events.

□  Readiness, liveness, and startup probes answer the right operational questions.

□  Operators can distinguish workload symptoms from cluster symptoms.

□  Alerting points toward an operator action or decision.

□  Retention is sufficient for the incidents the team is expected to investigate.

□  Material maintenance has documented prechecks.

□  Planned node maintenance uses a safe drain/eviction path appropriate to the environment.

□  Disruption budgets are reviewed for critical workloads.

□  Validation occurs before the next maintenance step.

□  Stop and rollback conditions are explicit.

□  Pending Pods are understood rather than ignored.

□  Resource requests are present and periodically reviewed for important workloads.

□  Actual usage is compared with requested capacity using the available metrics pipeline.

□  Placement constraints and failure-domain concentration are understood.

□  Autoscaling assumptions match request and scheduling behavior.

□  Critical workloads have a defensible replica and placement posture.

□  Planned disruption behavior has been reviewed separately from node-pressure behavior.

□  Probe configuration supports graceful traffic removal and appropriate recovery.

□  The team understands what happens when a node becomes unavailable or pressured.

□  Kubernetes support and patch posture are current for the environment.

□  Upgrade responsibilities and compatibility checks are documented.

□  Access and authorization are reviewed periodically.

□  Audit data availability and retention are known.

□  Security baseline checks are recurring, not one-time deployment tasks.

□  Backup ownership is explicit.

□  Cluster-state recovery mechanics are documented for the deployment model.

□  Application data recovery is accounted for separately where needed.

□  Recovery dependencies are documented.

□  Restore or recovery validation has a safe test path.

The output should not be "the cluster is healthy."

A useful review leaves you with:

1.  evidence for each of the six domains,

2.  a short list of unknowns that still need investigation,

3.  a prioritized operator backlog,

4.  a validation criterion for every remediation,

5.  a stop, rollback, or recovery condition for every material change.

If you only remember one thing, make it this: do not make Day-2 decisions from a single signal. Kubernetes reliability is the interaction of workload health, scheduling, disruption behavior, lifecycle maintenance, security controls, and recovery capability.

Your next action is to take one cluster, run the checklist, and refuse to turn an unknown into a guess. Capture the missing evidence as work. That alone will make the next maintenance window, incident, and upgrade easier to reason about.

•  Kubernetes Observability

•  Resource Management for Pods and Containers

•  Node Autoscaling

•  Liveness, Readiness, and Startup Probes

•  Safely Drain a Node

•  Node-pressure Eviction

•  Operating etcd clusters for Kubernetes

•  Kubernetes Security Checklist

•  Kubernetes Auditing

•  Kubernetes Releases