DEV Community: Illia Bilous

kubectl drain exits 0. Your database sits Pending.

Illia Bilous — Wed, 28 Oct 2026 21:21:41 +0000

kubectl drain has one job: evict the pods on a node, respecting disruption budgets. When every eviction is accepted, it exits 0. It does not check that the evicted pods have anywhere to go. That is the scheduler's problem, and the scheduler only gets asked after the pod is already gone.

Most of the time that is fine. Two cases where it isn't come up on almost every production cluster: a pod whose disk lives in one zone, and a pod that holds a port on the host. Both drain cleanly, and in both the replacement pod never starts.

Case 1: a zonal disk, and a zone with no room

A pod's PersistentVolume can carry a required nodeAffinity. The scheduler's VolumeBinding filter only places the pod on a node whose labels satisfy it. For a cloud disk the affinity names a zone, because the disk physically lives there.

The part that is easy to miss: CSI drivers write that zone in their own label key, not in topology.kubernetes.io/zone:

Driver	Key on the PV and on the node
Azure Disk	`topology.disk.csi.azure.com/zone`
Amazon EBS	`topology.ebs.csi.aws.com/zone`
GCE Persistent Disk	`topology.gke.io/zone`

So a script that looks for topology.kubernetes.io/zone on your PVs finds nothing and concludes the disks are not pinned. They are.

Now drain a node in that zone when the other nodes in the zone can't take the pod (a taint, a selector, no room). Nodes in other zones have room, but the disk can't follow. The drain exits 0, and the pod sits Pending.

Reproduce it on kind

Four nodes: a control plane, two workers in zone za (one tainted for another team), one worker in zone zb. The PV's affinity uses the Azure Disk key, the way a real Azure disk would.

kind create cluster --name drain-demo --config - <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes: [{role: control-plane}, {role: worker}, {role: worker}, {role: worker}]
EOF

for n in worker:za worker2:za worker3:zb; do
  kubectl label node drain-demo-${n%%:*} \
    topology.kubernetes.io/zone=${n##*:} topology.disk.csi.azure.com/zone=${n##*:}
done
kubectl taint node drain-demo-worker workload=db:NoSchedule

Then a PV pinned to za in the driver's key, and a one-replica StatefulSet that claims it:

apiVersion: v1
kind: PersistentVolume
metadata: {name: pv-db}
spec:
  capacity: {storage: 1Gi}
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  hostPath: {path: /tmp/pv-db, type: DirectoryOrCreate}
  claimRef: {namespace: data, name: data-db-0}
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - {key: topology.disk.csi.azure.com/zone, operator: In, values: [za]}

(The StatefulSet is a pause container with a volumeClaimTemplates entry named data in namespace data; full manifest at the end.)

db-0 lands on worker2, the only untainted node in za. Drain it:

$ kubectl drain drain-demo-worker2 --ignore-daemonsets --delete-emptydir-data
...
$ echo $?
0
$ kubectl -n data get pod db-0
NAME   READY   STATUS    RESTARTS   AGE
db-0   0/1     Pending   0          20s

and the scheduler's reason:

0/4 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 1 node(s) were unschedulable, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

worker3 has room, but it's in zb. worker is in za, but it's tainted. Nothing else is left.

Case 2: a host port, held on every other node

An ingress controller or a node-local agent is often run as a Deployment with hostPort (or hostNetwork: true), one replica per node. The scheduler's NodePorts filter won't put two pods holding the same host port on one node.

One detail makes this easy to miss: with hostNetwork: true, every containerPort you declare is a host port. The API server fills in hostPort for you.

Drain one of those nodes, and the evicted replica needs a node where the port is free. If every other eligible node already runs a replica, there isn't one:

$ kubectl -n edge get pods -o wide --no-headers | awk '{print $1,$3,$7}'
ingress-5bdbf6d67b-6gxqd Running drain-demo-worker3
ingress-5bdbf6d67b-8tdnz Running drain-demo-worker2

$ kubectl drain drain-demo-worker3 --ignore-daemonsets --delete-emptydir-data
...
$ echo $?
0
$ kubectl -n edge get pods
NAME                       READY   STATUS    RESTARTS   AGE
ingress-5bdbf6d67b-8tdnz   1/1     Running   0          22s
ingress-5bdbf6d67b-tjkvq   0/1     Pending   0          21s

0/4 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 1 node(s) were unschedulable, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/4 nodes are available: 1 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.

Checking before you drain

Both checks are mechanical. Before draining node N, for each pod on it:

Volumes. For each PVC the pod mounts, read the bound PV's spec.nodeAffinity.required. Find the nodes, other than N, whose labels satisfy it. Match on whatever key the PV uses, not just topology.kubernetes.io/zone.
Host ports. Collect the pod's host ports (hostPort, or every containerPort when hostNetwork: true). Drop every node where a running pod already holds one of them, by address, protocol and number.
The rest. Drop the nodes the pod's nodeSelector, required node affinity and tolerations rule out, and the cordoned ones.

If no node is left, the pod has nowhere to go, and the drain will still exit 0.

Disclosure, and a bug of my own

I build Shoalcheck, a read-only tool that answers this before you drain (shoalcheck preflight drain <node>, free on every plan). It got both cases wrong until this week:

Until v0.1.2 it read a volume's zone only from topology.kubernetes.io/zone, so a disk pinned in a CSI driver's key looked unpinned.
Until v0.1.3 it didn't read host ports at all.

On the cluster above, the old versions answered that nothing they check stops the drain. The new ones answer:

$ shoalcheck preflight drain drain-demo-worker3       # v0.1.2
✓ drain-demo-worker3: nothing this model checks stops the drain
$ echo $?
0

$ shoalcheck preflight drain drain-demo-worker3       # v0.1.3
✗ drain-demo-worker3 cannot be evacuated: 1 thing stops the drain
1. Losing node drain-demo-worker3 leaves 1 pod with no node this model accepts
   If node drain-demo-worker3 were unavailable, 1 pod running on it would have no remaining node in this reading that satisfies the constraints this model checks.
   · Snapshot taken at 2026-10-28T12:30:28Z. Each resource kind is listed independently, so the node, pod and budget views can differ by moments; re-verify against the live cluster before acting.
   · edge/ingress-5bdbf6d67b-6gxqd requests 10m cpu and 16Mi memory and host port 8080/TCP; no surviving node in this reading satisfies the modelled constraints
   · modelled here: …
   · not modelled: …
$ echo $?
2

The checks are still a subset of the scheduler. The tool says so in every answer, with the list of what it does not model: preemption, autoscaling, inter-pod affinity, topology spread, and more. It can tell you a drain will strand something. It never tells you a drain is safe.

The drain check is one of the things in this 74-second walkthrough, filmed on the real product (a kind test cluster, every frame a capture):

Full manifest

apiVersion: v1
kind: Namespace
metadata: {name: data}
---
apiVersion: v1
kind: PersistentVolume
metadata: {name: pv-db}
spec:
  capacity: {storage: 1Gi}
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  hostPath: {path: /tmp/pv-db, type: DirectoryOrCreate}
  claimRef: {namespace: data, name: data-db-0}
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - {key: topology.disk.csi.azure.com/zone, operator: In, values: [za]}
---
apiVersion: apps/v1
kind: StatefulSet
metadata: {name: db, namespace: data}
spec:
  replicas: 1
  serviceName: db
  selector: {matchLabels: {app: db}}
  template:
    metadata: {labels: {app: db}}
    spec:
      containers:
      - name: db
        image: registry.k8s.io/pause:3.10
        resources: {requests: {cpu: 10m, memory: 16Mi}}
        volumeMounts: [{name: data, mountPath: /data}]
  volumeClaimTemplates:
  - metadata: {name: data}
    spec:
      accessModes: [ReadWriteOnce]
      storageClassName: manual
      resources: {requests: {storage: 1Gi}}
---
apiVersion: v1
kind: Namespace
metadata: {name: edge}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: ingress, namespace: edge}
spec:
  replicas: 2
  selector: {matchLabels: {app: ingress}}
  template:
    metadata: {labels: {app: ingress}}
    spec:
      containers:
      - name: proxy
        image: registry.k8s.io/pause:3.10
        ports: [{containerPort: 8080, hostPort: 8080}]
        resources: {requests: {cpu: 10m, memory: 16Mi}}

Kubernetes upgrade preflight checklist for AKS, EKS and GKE

Illia Bilous — Mon, 26 Oct 2026 21:48:16 +0000

Managed Kubernetes makes the control-plane upgrade a button. The button does not tell you what will break in the workloads, and it does not tell you that the node-pool roll will stall at the first node with a bad PodDisruptionBudget. This is the checklist to run on a cluster (call it prod-eu-1) before you press it. The first four sections apply everywhere; the last covers what differs on AKS, EKS and GKE.

0. Read the release notes and pick a target

Upgrade one minor version at a time. Kubernetes does not support skipping minors on the control plane, and the managed providers enforce this in different ways. Read the upstream changelog for the target version, specifically the "Urgent upgrade notes" and "Deprecation" sections, and your provider's own release notes for that version.

1. APIs removed in the target version

Kubernetes removes beta API versions on a published schedule. The authoritative list is the upstream deprecated API migration guide at kubernetes.io. Some well-known removals:

1.22: extensions/v1beta1 and networking.k8s.io/v1beta1 Ingress, apiextensions.k8s.io/v1beta1 CRDs, admissionregistration.k8s.io/v1beta1 webhooks.
1.25: batch/v1beta1 CronJob, policy/v1beta1 PodDisruptionBudget and PodSecurityPolicy, autoscaling/v2beta1 HorizontalPodAutoscaler, discovery.k8s.io/v1beta1 EndpointSlice.
1.26: autoscaling/v2beta2 HorizontalPodAutoscaler, flowcontrol.apiserver.k8s.io/v1beta1.
1.27: storage.k8s.io/v1beta1 CSIStorageCapacity.
1.29: flowcontrol.apiserver.k8s.io/v1beta2.
1.32: flowcontrol.apiserver.k8s.io/v1beta3.

The list is illustrative; check the guide for your exact target.

Find what still uses a removed API

There are three places a removed API can hide, and none of them alone is sufficient.

Live requests. The API server counts calls to deprecated APIs. This is the best signal that something is actively using one:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Each series carries group, version, resource and removed_release labels. A non-zero series with removed_release="1.32" means a client called that API recently. Metrics reset on API server restart and on managed services you may not be able to scrape /metrics at all, so treat an empty result as "not seen", not "not used".

What the cluster still serves. Discovery lists every group/version the server answers:

kubectl api-versions | sort
kubectl api-resources --verbs=list -o name | sort

A beta group/version in this list that your target removes is the thing to chase.

What is stored and what is applied. Objects are stored in one version and served in others, so kubectl get returns the preferred version regardless of how it was written. Manifests are where the old version lives: grep Helm charts, Kustomize overlays, Terraform and CI templates.

grep -rEn "apiVersion: (extensions/v1beta1|policy/v1beta1|batch/v1beta1|autoscaling/v2beta1|autoscaling/v2beta2)" .
helm template my-release ./chart | grep -E "^apiVersion:" | sort | uniq -c

Also check what Helm has recorded: a release whose stored manifest uses a removed API can fail to upgrade even when the live objects are fine. The helm mapkubeapis plugin exists for that.

Finally, check admission webhooks and CRD conversion webhooks. A webhook registered with a removed admissionregistration version, or a CRD that declares a conversion webhook whose Service has no ready endpoints, will cause API errors that look unrelated to the upgrade.

2. PDBs that will stall node-pool upgrades

The control plane upgrade is quick. The node-pool upgrade is where clusters get stuck. Every provider's node upgrade drains each node, and drain respects PodDisruptionBudgets. A budget with zero allowed disruptions pins its pod, and the node waits.

kubectl get pdb -A
kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name)"'

Three causes cover most cases:

A Deployment with one replica and minAvailable: 1 (or maxUnavailable: 0). Allowed disruptions is always zero.
minAvailable equal to the replica count.
A workload that is already unhealthy, so the budget has nothing to give.

A roll with surge nodes does not fix this. Surge adds capacity so pods have somewhere to go; it does not change the budget. The pod still cannot be evicted. Providers handle the stall differently: some time out and fail the upgrade, some force the eviction after a window. See section 5, and read your provider's current documentation for the exact timeout.

Beyond budgets, a node roll also trips on:

pods with no controller, which kubectl drain refuses without --force;
pods with emptyDir data;
pods that cannot reschedule: node selectors, required anti-affinity, taints, zonal volumes, or no spare capacity once a node is gone.

The same checks apply as for a single drain; why kubectl drain hangs covers each. The difference for an upgrade is that you need the answer for every node in the pool, not one.

Also confirm the pool can absorb a surge node: VM quota for the family and free IP addresses in the subnet. A surge that cannot get either fails for reasons unrelated to Kubernetes.

3. Version skew

The upstream version skew policy sets the limits. Verify the current numbers against it, since they have changed over time:

The kubelet may not be newer than the API server. It may be older, by up to three minor versions on recent releases (it was two on older ones).
kubectl is supported within one minor version of the API server, in either direction.
Controller-manager, scheduler and cloud-controller-manager must not be newer than the API server.

On managed services the control plane moves first, so the practical rule is: after the control plane upgrade, the nodes will be behind, and you want that gap to stay inside the policy while you work through the pools. If your nodes are already two minors behind, upgrading the control plane another minor puts them at three, and then you have no room. Upgrade node pools soon after the control plane rather than letting them lag.

Check the gap:

kubectl get nodes -o custom-columns='NODE:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion'
kubectl version

Clients matter too: CI runners with an old kubectl, and operators built against client libraries that only speak a removed API version, break at the API server upgrade, not the node upgrade.

4. Add-ons

Add-ons are what actually break. Each one has its own compatibility matrix with Kubernetes versions, and most are not upgraded automatically.

Make a list:

kubectl get deploy,ds -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'

Then, for each one, look up whether the target Kubernetes version is supported by the add-on version you run. Typical entries:

CNI plugin and CSI drivers (storage drivers are the ones that hurt most when they are wrong).
CoreDNS and kube-proxy, where the provider manages versions.
Cluster autoscaler (its minor version should match the cluster's) and ingress controllers.
cert-manager, service meshes, policy engines and any operator that ships CRDs.
Anything with a webhook. A webhook with failurePolicy: Fail and no backing pod can block all writes of the resource it covers.

5. What to check per provider

These are the managed-service specifics that have been stable for a while. Confirm details against the current docs, because defaults and limits get changed.

EKS

Upgrade insights. EKS runs checks for deprecated API usage and other issues and reports them per cluster. In the console or with aws eks list-insights --cluster-name prod-eu-1.
Add-ons. Managed add-ons (VPC CNI, CoreDNS, kube-proxy, EBS CSI) have versions per Kubernetes release: aws eks describe-addon-versions --kubernetes-version <target> --addon-name <name>. Self-managed ones are your job.
Order. Control plane first, then add-ons, then node groups. A control plane upgrade cannot be rolled back.
Managed node groups. The update config has maxUnavailable (default one node at a time) or a percentage; raising it speeds up the roll but concentrates the disruption.

AKS

Available versions. az aks get-upgrades -g <rg> -n prod-eu-1 -o table. AKS does not let you skip minor versions.
Node surge. Each node pool has a max surge setting, with a default of a percentage of the pool; az aks nodepool update --max-surge 33% changes it. Surge nodes count against your VM quota and subnet addresses.
Drain behaviour. AKS drains nodes during upgrade and honours PDBs; a drain that cannot finish within the node drain timeout fails the upgrade. Check the current timeout in the docs and set it deliberately.

GKE

Release channels and maintenance windows. On a release channel the control plane upgrades on Google's schedule within your window; set exclusions for periods you cannot afford a change.
Deprecation insights. GKE reports usage of deprecated APIs for upcoming versions in the console, and can pause automatic upgrades when it sees usage.
Surge upgrades. Node pools use maxSurge and maxUnavailable; the defaults favour one surge node and no unavailable nodes. Blue-green is an alternative for pools that need to roll back quickly.
PDBs. GKE honours PodDisruptionBudgets during node upgrades for a bounded time, after which it proceeds. Do not count on that: a budget that waits an hour then gets bypassed is not a design.

6. The checklist

Target version chosen, one minor step, release notes read.
Removed APIs: metric checked, discovery checked, manifests and Helm releases grepped, webhooks checked.
PDBs: none with zero allowed disruptions on a workload you are about to roll.
Drain blockers: unmanaged pods, emptyDir, unschedulable replacements, per node.
Quota and IP headroom for surge.
Skew: kubelet gap, kubectl, CI clients.
Add-on compatibility for the target version, with an upgrade order.
Backups and a rollback plan: the control plane usually cannot be downgraded, so the plan is "fix forward" or "restore a new cluster".
Staging cluster upgraded first, on the same version path.

A tool for part of it

Shoalcheck is a read-only Kubernetes audit tool, a CLI and a desktop app for macOS, Windows and Linux, and it installs nothing in the cluster. shoalcheck preflight upgrade <version> answers one question: what stops the way to that version. It compares the group/versions the cluster still serves with a removal schedule taken from the upstream deprecation guide, and it reports the node pools that cannot roll because a drain stops on them. That covers sections 1 and 2 above, not skew or add-ons. Preflight is free on every plan: shoalcheck.com, downloads on GitHub releases.

Closing

Most failed managed upgrades are not Kubernetes bugs. They are a removed API that nobody grepped for, a PDB that can never be satisfied, a node pool that cannot get surge capacity, or an add-on three versions behind. Each of those is visible in the cluster before the upgrade starts.

kubectl drain stuck? Why drains hang on PodDisruptionBudgets, and how to check before you start

Illia Bilous — Mon, 26 Oct 2026 21:47:29 +0000

You cordon a node, run kubectl drain, and the terminal prints the same line every five seconds:

error when evicting pods/"api-7d9c6b8f5-x2k4q" -n payments (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Nothing is broken. Drain is doing what it was told: asking the API server to evict pods one at a time and waiting. A drain hangs for a small number of reasons, and almost all of them can be read from the cluster before you start. This post goes through them with commands you can run against any cluster.

What drain actually does

kubectl drain <node> first cordons the node (marks it unschedulable), then evicts each pod through the Eviction API. Eviction is not delete. The API server checks the pod's PodDisruptionBudget (PDB) and refuses with HTTP 429 if the eviction would take the workload below its budget. kubectl keeps retrying, which is the "hang".

Drain also refuses to start on some pods, and it can succeed on eviction while the pod then has nowhere to run. Those are three different failure classes:

Eviction is refused (PDB).
Drain refuses the pod up front (unmanaged pods, local storage, DaemonSets).
Eviction works, but the replacement pod stays Pending.

1. PDB math

List every budget and look at one column:

kubectl get pdb -A

NAMESPACE   NAME   MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
payments    api    1               N/A               0                     41d
payments    web    N/A             1                 1                     41d

ALLOWED DISRUPTIONS is the number of pods the Eviction API will let go right now. If it is 0 for a workload that has a pod on your node, that pod cannot be evicted and the drain waits.

To get just the blocking budgets:

kubectl get pdb -A -o json | jq -r '
  .items[] | select(.status.disruptionsAllowed == 0)
  | "\(.metadata.namespace)/\(.metadata.name)  healthy=\(.status.currentHealthy)/\(.status.desiredHealthy) expected=\(.status.expectedPods)"'

minAvailable vs maxUnavailable

Both express the same idea from opposite sides.

minAvailable: N means at least N pods must stay healthy. Allowed disruptions = healthy pods minus N.
maxUnavailable: N means at most N pods may be unavailable. Allowed disruptions = N minus pods already unavailable.

Percentages are rounded up. With 10 replicas, minAvailable: 80% allows 2 disruptions; maxUnavailable: 25% allows 3 (2.5 rounds up).

The two behave differently when replicas change. minAvailable: 3 on a 3-replica Deployment allows zero disruptions forever; maxUnavailable: 1 keeps allowing one disruption at any size. If you want a budget that scales with the workload, prefer maxUnavailable.

A budget also counts only healthy pods. If one of three replicas is already crash-looping and the budget is minAvailable: 2, allowed disruptions is 0 even though no one touched anything. A PDB can set unhealthyPodEvictionPolicy: AlwaysAllow (beta since 1.27, GA in 1.31) so that pods which are not ready can be evicted regardless of the budget; check your cluster version before relying on it.

The single-replica Deployment with a PDB

This is the most common self-inflicted drain block:

apiVersion: apps/v1
kind: Deployment
metadata: { name: report-worker, namespace: payments }
spec:
  replicas: 1
  selector: { matchLabels: { app: report-worker } }
  template:
    metadata: { labels: { app: report-worker } }
    spec:
      containers: [{ name: app, image: registry.example/report-worker:1.4 }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: report-worker, namespace: payments }
spec:
  minAvailable: 1
  selector: { matchLabels: { app: report-worker } }

One replica, one required. Allowed disruptions is 0, always. maxUnavailable: 0 has the same effect. Every drain that touches the node hosting this pod will hang until someone deletes the budget or scales up.

The honest options are: run at least two replicas, set maxUnavailable: 1 and accept a brief gap, or drop the PDB for a workload that cannot meet it anyway. A PDB that can never be satisfied protects nothing and blocks every node upgrade.

Also check the selector. A PDB whose selector matches no pods shows expectedPods: 0, and a selector that matches pods from several controllers produces unspecified results; the API documentation says to keep one PDB per workload.

2. Pods that drain refuses to evict

Before any eviction, kubectl drain checks each pod and aborts with a message if it finds one of these. These are flags you have to choose deliberately.

Pods without a controller

A bare pod (no ReplicaSet, StatefulSet, Job or similar owner) will not be recreated anywhere. Drain refuses unless you pass --force, and then the pod is simply gone.

Find them:

kubectl get pods -A -o json | jq -r '
  .items[] | select((.metadata.ownerReferences // []) | length == 0)
  | "\(.spec.nodeName)\t\(.metadata.namespace)/\(.metadata.name)"'

Static (mirror) pods show up here too; those are owned by the node and drain ignores them. Anything else on the list is a pod you need to decide about: recreate it as a Deployment, or accept losing it.

emptyDir and local storage

Pods with an emptyDir volume lose that data on eviction. Drain refuses unless you pass --delete-emptydir-data (older versions called it --delete-local-data). That flag is a decision, not a fix. If the emptyDir holds a cache, fine. If it holds the only copy of something, it is not.

kubectl get pods -A -o json | jq -r '
  .items[] | select(any(.spec.volumes[]?; has("emptyDir")))
  | "\(.spec.nodeName)\t\(.metadata.namespace)/\(.metadata.name)"'

A different local-storage trap: a PersistentVolume with node affinity (local PVs, or zonal disks) pins the pod to a node or zone. The pod can be evicted, but the replacement will not schedule anywhere else. That belongs in section 3.

DaemonSets

DaemonSet pods are recreated on the same node immediately, so evicting them is pointless. Drain refuses until you pass --ignore-daemonsets, which skips them. This is the one flag nearly everyone passes:

kubectl drain prod-eu-1-node-a --ignore-daemonsets --delete-emptydir-data --timeout=10m

Put --timeout on every drain you script. Without it, a PDB block waits forever.

3. Pods that cannot reschedule

Eviction succeeds, the drain moves on, and the replacement pod sits in Pending. For a single-replica workload that is an outage; for a multi-replica one it is lost capacity. Check this before you drain, because the scheduler will not tell you until it is too late.

Look at what is already pending:

kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod -n payments <pod> | sed -n '/Events:/,$p'

The event reads like 0/6 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 Insufficient cpu, 1 node(s) had untolerated taint. Each clause is a reason, and you can test for each one ahead of time.

Node selectors and affinity. List what the pods on your node require:

kubectl get pods -A --field-selector spec.nodeName=prod-eu-1-node-a -o json | jq -r '
  .items[] | "\(.metadata.namespace)/\(.metadata.name)\t\(.spec.nodeSelector // {})\t\(.spec.affinity != null)"'

If a pod selects disktype=ssd and this is the only node with that label, there is nowhere for it to go. The same applies to required pod anti-affinity: a pod that must not share a node with its siblings needs one free node per replica.

Taints. Compare node taints to the tolerations of the pods you are about to move:

kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key'

Capacity. Look at what the remaining nodes can absorb:

kubectl describe nodes | grep -A6 "Allocated resources"
kubectl top nodes

Requests, not usage, decide scheduling. A node that looks idle in top can be full in Allocated resources.

Zones and volumes. A pod bound to a zonal volume can only restart in that zone. If the zone has no spare capacity, it stays Pending.

None of this replaces a real drain in a staging cluster, and the scheduler has more inputs than any manual check covers (topology spread constraints, priorities, extended resources). But the checks above catch the usual causes.

A pre-drain checklist

Run these in order:

kubectl get pdb -A and look for ALLOWED DISRUPTIONS of 0 on workloads with pods on the node.
List unmanaged pods and emptyDir pods on the node, and decide on --force and --delete-emptydir-data knowingly.
For each workload on the node, confirm it has a place to go: labels, taints, free requests, zone.
Cordon first (kubectl cordon), then drain with --ignore-daemonsets --timeout.
If it hangs, read the retry message. It names the pod and the namespace.

Doing this by hand for one node is fine. Doing it for every node in a pool before an upgrade is tedious and easy to get wrong.

Doing the same check with a tool

Shoalcheck is a read-only Kubernetes audit tool (CLI and desktop app) that installs nothing in the cluster. shoalcheck preflight drain <node> answers this one question: whether that node can be evacuated, and if not, which object stops it (a budget at zero, a pod nothing recreates, a volume pinned to the node, work with nowhere to go). It exits 0 when nothing stops the drain, 2 when something does, and 1 when RBAC refused some reads and the answer is partial. Preflight is free on every plan: shoalcheck.com, downloads on GitHub releases. Its model checks less than the scheduler does, so treat a clear result as "nothing found", not a guarantee.

Summary

A drain hangs because an eviction is refused or a replacement cannot start. A PDB with ALLOWED DISRUPTIONS of 0 is the usual cause, and the single-replica Deployment with minAvailable: 1 is the usual way to create one. Bare pods, emptyDir data and DaemonSets are things drain makes you acknowledge with flags. Unschedulable replacements are the quietest failure, so look at selectors, taints, capacity and zones before you cordon, not after.