DEV Community: evan

why your pod memory limits keep crashing and how to fix them

evan — Fri, 23 Oct 2026 11:42:26 +0000

you've probably seen a pod go into CrashLoopBackOff after a sudden traffic spike, even though the container's memory request matches the node's capacity. the root cause is an invisible cgroup pressure that the scheduler can't see, and a missing hard limit that lets the process eat the node's RAM until the kernel OOM killer steps in.

the hidden pressure

kubernetes only schedules based on the request field; if you omit a hard limit, the container can exceed its request without any back‑pressure. the node's cgroup will then balloon, starving sibling pods and eventually triggering a node‑wide OOM. the failure surface is hard to diagnose because the pod logs show nothing unusual—only the termination reason "OOMKilled".

apiVersion: v1
kind: Pod
metadata:
  name: memory‑stress
spec:
  containers:
  - name: app
    image: myorg/app:latest
    resources:
      requests:
        memory: "256Mi"
      limits:
        memory: "512Mi"   # explicit hard limit
    env:
    - name: ENABLE_METRICS
      value: "true"
  - name: cgroup‑exporter
    image: prom/cgroup-exporter:latest
    resources:
      limits:
        memory: "64Mi"
    volumeMounts:
    - name: cgroup
      mountPath: /host/sys/fs/cgroup
  volumes:
  - name: cgroup
    hostPath:
      path: /sys/fs/cgroup

the sidecar above scrapes /sys/fs/cgroup and pushes memory usage into Prometheus, giving you a real‑time view of the container's actual consumption versus its limits. coupling this with an alert on container_memory_working_set_bytes approaching container_spec_memory_limit_bytes catches overshoots before the kernel kills the pod.

trade‑offs and failure modes

limit too low – the pod throttles under normal load, increasing latency and causing request retries that amplify load. you need to benchmark realistic burst patterns and set the limit a safe headroom (often 1.5‑2× the 95th‑percentile usage).
limit too high – you re‑introduce noisy‑neighbor risk; a memory leak in one tenant can starve the rest of the node. run chaos experiments that inject a synthetic leak (stress --vm 2 --vm-bytes 300M) to verify bulkhead isolation.
metrics lag – if the exporter polls too infrequently, you may react after the OOM event. configure a scrape interval of ≤15s for critical services.
node‑level cgroup fragmentation – on long‑running clusters, leftover cgroup hierarchies can corrupt accounting. schedule a periodic systemd-run --quiet --property=Delegate=yes cleanup job to prune orphaned groups.

by treating the memory limit as a contract rather than a suggestion, you turn an invisible failure domain into an observable one. the contract must be continuously validated in production; otherwise you inherit the same brittle behavior you tried to avoid.

what's your approach to validating pod memory limits in production without triggering OOM kills

Why explicit cgroup limits are non‑negotiable in production Kubernetes clusters

evan — Tue, 20 Oct 2026 02:51:58 +0000

when a new tenant lands on a shared node, the first thing that goes wrong is usually a runaway container that steals memory or CPU, starving the rest of the workloads. in my last on‑call rotation the node went from 70 % to 100 % memory within minutes, the OOM killer started terminating unrelated pods, and the incident stretched beyond our SLA.

the missing guard: cgroup limits in pod specs

Kubernetes lets you describe resource requests and limits, but the defaults are often left at "0" (unbounded) because the UI hides the fields or the helm chart skips them for simplicity. without explicit memory and cpu limits, the container runs directly against the node’s cgroup hierarchy, which is exactly the noisy‑neighbor scenario we try to avoid.

apiVersion: v1
kind: Pod
metadata:
  name: data‑processor
spec:
  containers:
  - name: worker
    image: myorg/processor:latest
    resources:
      requests:
        cpu: "500m"
        memory: "256Mi"
      limits:
        cpu: "1"
        memory: "512Mi"
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://otel-collector:4317"

The limits line creates a hard cgroup cap; the kernel will throttle CPU and trigger OOM kill only for that container. note that the requests field also informs the scheduler, keeping the pod on a node with sufficient headroom.

trade‑offs you have to accept

throughput vs safety – a tightly capped pod may hit back‑pressure sooner, requiring you to tune your application’s internal queueing or enable graceful degradation.
cost impact – over‑provisioning limits forces you to allocate more nodes to meet aggregate demand, but the alternative is unpredictable eviction and higher incident toil.
observability burden – you now need to surface cgroup metrics (e.g. container_cpu_cfs_throttled_seconds_total, container_memory_working_set_bytes) into Prometheus and set alerts for sustained throttling or OOM events.

continuous validation with chaos engineering

Static limits are only as good as the workload profiles you measured. embed a simple chaos test in your CI pipeline:

func TestMemoryThrottle(t *testing.T) {
    // launch pod with 128Mi limit, feed 200Mi of data
    // assert that the process respects context cancellation after 30s
    // verify that the node’s memory usage stays below 85% during the run
}

Running this test on every PR forces developers to keep the container footprint in check and surfaces regression before it hits production. combine it with a production‑grade chaos runner (e.g. Litmus or Gremlin) that randomly spikes CPU on a node to verify that other pods stay within their cgroup caps.

edge cases to watch

burstable QoS – pods with requests < limits can be throttled under node pressure; if your SLAs depend on consistent latency, consider requests == limits to lock them in the Guaranteed QoS class.
sidecar interference – a logging sidecar that writes massive buffers can bypass the main container’s limits; enforce limits at the pod level or run sidecars in separate pods.
node‑level cgroup overrides – some managed K8s providers inject default limits that conflict with your spec; always inspect the effective cgroup hierarchy (/sys/fs/cgroup/...) on a test node.

by treating cgroup limits as a non‑negotiable contract rather than an optional knob, you shrink the failure surface dramatically and make chaos tests meaningful. how do you enforce and validate resource caps in your clusters, and what observability patterns have saved you from noisy‑neighbor incidents?

hard‑capped CPU in k8s: stopping silent throttling before it crashes your node

evan — Thu, 15 Oct 2026 10:22:02 +0000

in a typical autoscaling deployment you’ll see pods spin up, hit their request rates, and then the node suddenly starts paging. the symptom is a spike in latency without any OOM kills – a classic silent CPU throttle that the kernel reports only in cgroup stats.

the silent throttle

kubernetes lets you specify requests and limits, but the default cpu limit is soft when you use the cpu shares model on older cgroup v1 setups. the scheduler assumes the node can overcommit, and the kernel throttles containers that exceed their share without raising an alert. the result is a cascade: one noisy neighbor starves others, the node’s load average climbs, and latency budgets break.

a reproducible pod spec

apiVersion: v1
kind: Pod
metadata:
  name: throttle-demo
spec:
  containers:
  - name: app
    image: nginx:stable-alpine
    resources:
      requests:
        cpu: "250m"
      limits:
        cpu: "500m"   # hard cap with cgroup v2
    env:
    - name: LOAD_DURATION
      value: "30"
  - name: load-generator
    image: busybox
    command: ["sh", "-c", "while true; do dd if=/dev/zero of=/dev/null bs=1M count=10; sleep 1; done"]
    resources:
      limits:
        cpu: "500m"

the limits line forces a hard quota when the node runs a recent kernel with cgroup v2 enabled. you can verify the enforcement with kubectl top pod throttle-demo and cat /sys/fs/cgroup/cpu/kubepods/.../cpu.max.

trade‑offs to watch

over‑provisioning vs latency – setting limits too low guarantees throttling under normal load; too high defeats the purpose and re‑introduces noisy‑neighbor risk.
cost impact – hard caps improve bin‑packing efficiency, but you may need more nodes to meet peak demand, increasing egress and instance fees.
observability gap – most dashboards surface CPU usage but not throttling time. instrument your pods with cpu.throttled.time from /sys/fs/cgroup/cpu.stat and forward it to Prometheus; set alerts on a threshold of >5% throttled time over a 5‑minute window.
failure mode testing – simulate a burst with a temporary load-generator sidecar or a chaos‑engineered pod that spikes CPU. verify that the throttled pod degrades gracefully (e.g., returns 503 with a Retry‑After) rather than timing out.

cost‑aware scaling

pair hard caps with a custom HPA that uses the throttling metric as a scaling signal instead of pure CPU utilization. this prevents the controller from scaling up based on a single hot pod that’s already hitting its quota.

how do you validate that your resource limits actually protect the node under load without throttling critical paths

Why Unbounded Worker Pools Blow Up Go Services Under Back‑pressure

evan — Mon, 12 Oct 2026 15:59:25 +0000

problem statement

in a recent deployment we added a background worker pool to a high‑traffic HTTP service. the pool pulled jobs from a shared channel without any limit on the number of goroutines. under normal load the service seemed fine, but a spike in upstream traffic caused the request queue to fill, the pool spawned thousands of goroutines, and the pod hit OOM kills within seconds. the crash cascaded to the entire deployment because the readiness probe never recovered.

why the pool exploded

no back‑pressure – a plain chan Job without a buffer or a semaphore allows any number of producers to push work.
missing context propagation – workers ignored the request context, so they kept processing even after the client timed out.
absence of telemetry – only occasional log lines were emitted; the CPU and memory usage spiked unnoticed until the pod was evicted.

a minimal reproducible snippet

type Job struct{ payload []byte }

func startWorkerPool(jobs <-chan Job) {
    for job := range jobs { // <- unbounded loop
        go func(j Job) {
            // heavy processing, no ctx
            _ = process(j.payload)
        }(job)
    }
}

func main() {
    jobs := make(chan Job) // unbuffered, producers block only on send
    go startWorkerPool(jobs)
    http.HandleFunc("/enqueue", func(w http.ResponseWriter, r *http.Request) {
        // simulate burst of 10k requests per second
        for i := 0; i < 1000; i++ {
            jobs <- Job{payload: []byte("data")}
        }
        w.WriteHeader(http.StatusAccepted)
    })
    http.ListenAndServe(":8080", nil)
}

production‑grade fixes

bulkhead with semaphore – limit concurrent workers to a sane number (e.g., runtime.GOMAXPROCS()*2).
bounded channel or work‑queue – use a buffered channel and drop or back‑pressure when the buffer is full.
context‑aware workers – pass r.Context() downstream; abort processing when the client disconnects.
metrics & alerts – expose goroutine, heap, and queue length via Prometheus; set alerts on 80% memory usage.
chaos testing – inject artificial latency on the upstream API and verify that the pool respects the limits without OOM.

trade‑offs to consider

throughput vs latency – a tighter semaphore reduces peak throughput but guarantees latency bounds and prevents node eviction.
queue size – a small buffer protects the node but may increase request rejection rates; choose a size based on expected burst and acceptable 503 rate.
cost of retries – if you drop jobs, ensure idempotent downstream processing or a durable retry store (e.g., Kafka) to avoid data loss.
observability overhead – adding per‑request tracing adds CPU, but without it you cannot see the back‑pressure chain.

conclusion

unbounded worker pools are a classic hidden failure mode that surfaces only under load spikes. limiting concurrency, propagating cancellation, and instrumenting the queue are non‑negotiable in production. how does your stack enforce bulkhead isolation for background processing, and what metrics do you rely on to detect a runaway goroutine storm?