# Silent Node Killer: How Mem-Backed emptyDir Vols. degraded our k8s Perf.  Cluster

![AI generated](https://cdn.hashnode.com/uploads/covers/6a25ff63980616b1c020fc7a/c2ae8162-0f45-4d57-ab12-baf63ce7e6e7.png align="center")

When you operate an on-prem K8s platform with a static node count that hasn't changed for months, a single dead node could have a ripple effect on the entire cluster, and the workloads being run causing a massive headache. Unlike the cloud managed K8s clusters, at tmies we don't have the luxury of auto-scaling groups silently spinning up fresh instances in the background. If a node drops, we lose hard capacity until someone physically intervenes ,and in case the clsuter is already under duress it only amplifies.

This is exactly what was experienced on the primary ***Perf*** cluster very recently. One of the dev teams deployed a distributed workload for QA with several enhancements including using memory backed `emptyDir` vols. as a ephemeral cache, which resulted in a cascading failure where the cluster performance was significantly degraded. There was no `NotReady` transition or graceful pod eviction - instantaneous lockups. Couldn't even SSH into some of the boxes.

Here is how a single volume mount config exploited a K8s accounting loophole (Issue #119611) to starve our cluster, and the logical steps we took to debug the app.

### The Investigation: Step-by-Step

**Step 1: The Hard Reset and Kernel Autopsy** Since `sshd` was completely unresponsive, we had to hit the nodes via out-of-band mgmt to issue a hard power cycle. Once the node rejoined the cluster, my first instinct was to query [Loki (via Grafana)](https://grafana.com/oss/loki/) for the `kubelet` logs just prior to the crash. We fully expected to see the standard K8s resource starvation chain-of-events -

*   `kubelet` crossing its `evictionHard` threshold
    
*   tainting node with `MemoryPressure`, and
    
*   gracefully terminating `BestEffort` or `Burstable` pods first
    

Instead, the Loki streams had completely dropped the last few minutes before the crash—a classic indicator that the logging daemon itself was starved of CPU/memory. Pulling `dmesg` and `journalctl -k` directly from the local nodes revealed system-level chaos - kernel's OOMkiller had been invoked, but it wasn't just reaping misbehaving containers; it was indiscriminately killing core OS processes. The node didn't gracefully handle the memory pressure; the `kubelet` was completely ambushed before it could even calculate the usage and trigger an eviction loop.

**Step 2: Isolating the Suspect Workload** As the load-testing event was already previously advertised before hand we knew where the smoking gun might be pointing at. The workload deployment manifest included:

```yaml
volumes:
  - name: cache-vol
    emptyDir:
      medium: Memory
```

**Step 3: The Shared Quota Trap** An `emptyDir` with `medium: Memory` tells K8s to mount a `tmpfs` (RAM disk) inside the container. Here is the critical detail that caught me off guard: `tmpfs` **usage counts directly toward your container's mem** [**limit**](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-limits-are-run)**.**

> *   The memory limit for the Pod or container can also apply to pages in memory backed volumes, such as an `emptyDir`. The kubelet tracks `tmpfs` emptyDir volumes as container memory use, rather than as local [ephemeral storage](https://kubernetes.io/docs/concepts/storage/ephemeral-storage/). When using memory backed `emptyDir`, be sure to check the notes [b](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#memory-backed-emptydir)elow
>     

It acts as a shared quota. FOr example, If you assign a container a mem limit of `1Gi`, that must accommodate both the app's RSS/heap *and* any files written to the `tmpfs` volume. If the sum of the app mem and the `tmpfs` data exceeds the defined limit, the container might get OOMKilled.

**Step 4: Discovering the Cgroup Loophole (The Root Cause)** The devs hadn't set a `sizeLimit` on the volume. As the app was aggressively writing cache files, the shared quota was quickly breached, and the app was OOMKilled. Standard K8s behavior, right?

But here is where the K8s accounting loophole triggered the outage - when a container is OOMKilled and restarts, the pod remains on the node, and the data previously written to the `tmpfs` *persists*. However, when the new container spins up, the Linux kernel (possibly) often fails to account for that pre-existing `tmpfs` data in the *new* container's cgroup.

This data becomes "ghost mem." It consumes RAM but is invisible to the new cgroup limit. The app restarts, writes more files to the `tmpfs`, consumes its full mem limit again, crashes, and restarts. This loop continued until the entire node ran out of RAM and locked up. Now how did the `kubelet` not step in earlier to stop this madness is beyond me.

### Learnings

K8s is incredibly powerful, but sometimes it expects you to provide strict guardrails. When you don't, features like mem-backed volumes become loaded weapons.

*   It is advisable to enforce size limits for `emptyDir` volumes in new workloads (pods) using a policy mechanism such as [ValidationAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/).
    

*   **Set a** `sizeLimit` **on mem-backed volumes.** If you use `medium: Memory`, cap it. If a volume hits its `sizeLimit`, the `kubelet` gracefully evicts the pod instead of triggering an abrupt OOMKill.
    

```yaml
emptyDir:
  medium: Memory
  sizeLimit: 500Mi

```

*   **Enforce hard mem limits.** Set `resources.limits.memory` on the pod to ensure the `kubelet` and container runtime have a hard ceiling to enforce.
    
*   **Use Resourcequotas.** At the platform level, enforce a `ResourceQuota` in every ns that mandates limits. This prevents teams from deploying unbounded apps in the first place.
    
*   **DON'T Rely on** `LimitRange` **for** `emptyDir` **sizing.** While a `LimitRange` might apply default sizes to containers, it does not effectively prevent the mem overcommit scenario caused by unbound `tmpfs` volumes.
    
*   **DON'T Treat** `tmpfs` **like disk storage.** Educate the devs that `medium: Memory` is RAM. It shares the same budget as the app's heap size. If an app needs a massive scratchpad, write to standard disk-backed storage instead.