When you operate an on-prem K8s platform with a static node count that hasn't changed for months, a single dead node could have a ripple effect on the entire cluster, and the workloads being run causing a massive headache. Unlike the cloud managed K8s clusters, at tmies we don't have the luxury of auto-scaling groups silently spinning up fresh instances in the background. If a node drops, we lose hard capacity until someone physically intervenes ,and in case the clsuter is already under duress it only amplifies.
This is exactly what was experienced on the primary Perf cluster very recently. One of the dev teams deployed a distributed workload for QA with several enhancements including using memory backed emptyDir vols. as a ephemeral cache, which resulted in a cascading failure where the cluster performance was significantly degraded. There was no NotReady transition or graceful pod eviction - instantaneous lockups. Couldn't even SSH into some of the boxes.
Here is how a single volume mount config exploited a K8s accounting loophole (Issue #119611) to starve our cluster, and the logical steps we took to debug the app.
The Investigation: Step-by-Step
Step 1: The Hard Reset and Kernel Autopsy Since sshd was completely unresponsive, we had to hit the nodes via out-of-band mgmt to issue a hard power cycle. Once the node rejoined the cluster, my first instinct was to query Loki (via Grafana) for the kubelet logs just prior to the crash. We fully expected to see the standard K8s resource starvation chain-of-events -
kubelet crossing its evictionHard threshold
tainting node with MemoryPressure, and
gracefully terminating BestEffort or Burstable pods first
Instead, the Loki streams had completely dropped the last few minutes before the crash—a classic indicator that the logging daemon itself was starved of CPU/memory. Pulling dmesg and journalctl -k directly from the local nodes revealed system-level chaos - kernel's OOMkiller had been invoked, but it wasn't just reaping misbehaving containers; it was indiscriminately killing core OS processes. The node didn't gracefully handle the memory pressure; the kubelet was completely ambushed before it could even calculate the usage and trigger an eviction loop.
Step 2: Isolating the Suspect Workload As the load-testing event was already previously advertised before hand we knew where the smoking gun might be pointing at. The workload deployment manifest included:
volumes:
- name: cache-vol
emptyDir:
medium: Memory
Step 3: The Shared Quota Trap An emptyDir with medium: Memory tells K8s to mount a tmpfs (RAM disk) inside the container. Here is the critical detail that caught me off guard: tmpfs usage counts directly toward your container's mem limit.
- The memory limit for the Pod or container can also apply to pages in memory backed volumes, such as an
emptyDir. The kubelet tracks tmpfs emptyDir volumes as container memory use, rather than as local ephemeral storage. When using memory backed emptyDir, be sure to check the notes below
It acts as a shared quota. FOr example, If you assign a container a mem limit of 1Gi, that must accommodate both the app's RSS/heap and any files written to the tmpfs volume. If the sum of the app mem and the tmpfs data exceeds the defined limit, the container might get OOMKilled.
Step 4: Discovering the Cgroup Loophole (The Root Cause) The devs hadn't set a sizeLimit on the volume. As the app was aggressively writing cache files, the shared quota was quickly breached, and the app was OOMKilled. Standard K8s behavior, right?
But here is where the K8s accounting loophole triggered the outage - when a container is OOMKilled and restarts, the pod remains on the node, and the data previously written to the tmpfs persists. However, when the new container spins up, the Linux kernel (possibly) often fails to account for that pre-existing tmpfs data in the new container's cgroup.
This data becomes "ghost mem." It consumes RAM but is invisible to the new cgroup limit. The app restarts, writes more files to the tmpfs, consumes its full mem limit again, crashes, and restarts. This loop continued until the entire node ran out of RAM and locked up. Now how did the kubelet not step in earlier to stop this madness is beyond me.
Learnings
K8s is incredibly powerful, but sometimes it expects you to provide strict guardrails. When you don't, features like mem-backed volumes become loaded weapons.
It is advisable to enforce size limits for emptyDir volumes in new workloads (pods) using a policy mechanism such as ValidationAdmissionPolicy.
Set a sizeLimit on mem-backed volumes. If you use medium: Memory, cap it. If a volume hits its sizeLimit, the kubelet gracefully evicts the pod instead of triggering an abrupt OOMKill.
emptyDir:
medium: Memory
sizeLimit: 500Mi
Enforce hard mem limits. Set resources.limits.memory on the pod to ensure the kubelet and container runtime have a hard ceiling to enforce.
Use Resourcequotas. At the platform level, enforce a ResourceQuota in every ns that mandates limits. This prevents teams from deploying unbounded apps in the first place.
DON'T Rely on LimitRange for emptyDir sizing. While a LimitRange might apply default sizes to containers, it does not effectively prevent the mem overcommit scenario caused by unbound tmpfs volumes.
DON'T Treat tmpfs like disk storage. Educate the devs that medium: Memory is RAM. It shares the same budget as the app's heap size. If an app needs a massive scratchpad, write to standard disk-backed storage instead.
