One morning our production k8s cluster started acting strange. pods were dying at random. Not in one app, not in one ns - different pods, in different places, going down and coming back, again and again and again. No new deploy had gone out. Nothing in any of the app logs explained it. For a while it just looked like pure, unexplainable chaos.
The strange part: running pods do not need etcd
Before the cause, one fact that our team (half, atleast) was well aware is worth saying first - a pod that is already running does not talk to etcd. It does not even talk to the API server to keep running. Once the kubelet has started a container, the app just runs and serves traffic. You can take the whole control plane offline and your running pods will keep working.
This knowledge precisely made things a lot harder to debug.
The cause: a slow disk breaks the heartbeat of the cluster
etcd is the database that holds all k8s state. It is built for strong consistency, and it pays for that with one strict rule: every write must be saved to disk before it counts. Technically, etcd writes to a write-ahead log (WAL) and calls fsync to force the bytes onto the disk. Only after that fsync returns does the write commit.
etcd also runs as a small cluster of members that agree using the Raft algorithm. Every write goes through the leader and must be confirmed by a quorum, and each member must fsync it. This means one thing that becomes very important later: if the leader's disk is slow, every write in the whole cluster slows down at once.
Now connect this to pods. Many parts of k8s stay alive by renewing a "lease" before a deadline. A lease is just a small record that says "I am still here," and renewing it is a write - which means a write to etcd, which means an fsync. There are two leases that matter here:
Leader election leases. Control-plane components like the scheduler, the controller-manager, and many operators run more than one copy, but only one is active at a time. The active one must renew its leader lease every few seconds. If it cannot renew in time, it can no longer prove it is the leader. These programs are written to do the safe thing in that case: they exit on purpose, so that two leaders never act at once.
Node leases. Each node's kubelet renews a node lease about every 10 seconds to say "this node is alive." If those renewals stop arriving, the control plane marks the node NotReady, and after a grace period it starts evicting the pods on it.
Here is the full chain in one line:
disk I/O stall → fsync is slow → etcd WAL commit is slow → Raft writes back up → leases cannot renew in time → leader components exit and nodes go NotReady → pods restart or get evicted.
That is why our pods "died." A leader-elected pod killed itself because it could not renew its lease in time. An ordinary pod was evicted because its node missed too many heartbeats. In both cases, the thing that failed was a small write that could not fsync through a disk-stalled etcd. Our disk had short latency spikes every few minutes, and those spikes were printing themselves onto the pod crash schedule.
One more detail made it worse. The API server's own health check includes an etcd check. When etcd was slow, the API server itself started returning errors. So a single slow disk did not fail gently - it knocked over the one dependency that every lease and heartbeat runs through, all at the same time.
The debugging: looking in the wrong place first
The hard part of this kind of incident is that the symptom and the cause are far apart. We lost a few hours looking at the wrong layers. Here is the path we took, including the dead ends, because the dead ends are useful.
First we blamed the apps. The pods that died had no pattern at the app level. Different images, different teams, different ns. That told us it was not an app bug. If it were, it would hit one app, not all of them.
Then we blamed networking and resources. We checked CPU and memory limits, OOM kills, and network policies. Nothing lined up. The crashes did not match any resource graph.
Then we saw etcd's metrics rising - and talked ourselves out of it. Somewhere in here we did glance at etcd. The fsync and WAL commit times were climbing, But we waved it off. etcd sits far below the apps in the stack, and the pods that were dying were many links away from it. so we filed the etcd numbers under "probably related but unrelated" and kept looking elsewhere.
Then we noticed the timing. The crashes came in waves, every five to ten minutes. Random pods, but a regular beat. Random-but-periodic is a strong clue. It means something shared and low-level is failing on a cycle, not one app misbehaving.
Then we read the control-plane logs. This is where it turned. The API server logs showed timeouts talking to etcd. The etcd logs showed leader elections happening again and again. A healthy etcd almost never changes leader. Frequent leader elections almost always mean the disk or the network is too slow.
Then we went back to the etcd metrics we had dismissed. Once the logs pointed at etcd, we stopped waving off those numbers and actually read them. Two metrics tell the story directly:
etcd_disk_wal_fsync_duration_seconds etcd_disk_backend_commit_duration_seconds
On a healthy cluster the p99 of these should be a few milliseconds. Ours were spiking past 100ms, again on that same five-to-ten-minute beat. A good rule of thumb: if p99 of the WAL fsync is consistently above 100ms, you have a storage problem, not a k8s problem. These were the same numbers we had seen an hour earlier and ignored — only this time we let ourselves believe them.
Then we confirmed it at the disk as the metrics showed high latency and a disk pinned near 100% busy, in the same five-to-ten-minute waves. We could also see other workloads driving I/O on that same device at those exact moments. That was the final answer: etcd was sharing a disk with noisy I/O, and that disk could not keep fsync fast and steady.
The resolution: give etcd a disk it does not have to share
The fix was not in k8s settings. It was in storage.
The root problem was that etcd's disk was shared with other heavy I/O. So our first and most important change was simple: give etcd its own dedicated, fast, low-latency disk - a local NVMe SSD - and keep all other heavy I/O off it. No container storage, no logs, no batch jobs on the same device.
There are storage tricks that make fsync return instantly by not really waiting for the write to land (for example, disabling sync writes at the filesystem). These make the symptom disappear, but they break etcd's one job: durability.
With a rolling upgrade we moved the entire etcd cluster to dedicated NVMe, and immeidately witnessed the fsync p99 dropped back to a few milliseconds, the leader elections stopped, and the random pod crashes stopped with them.
The action: so it never surprises us again
Fixing the disk ended the fire. These changes make sure it does not start again, and that if it does, we see it early.
Alert on the real metric. We now alert when p99 of etcd_disk_wal_fsync_duration_seconds goes above a safe threshold. This catches the problem while etcd is still slow, before it becomes pod chaos.
Watch leader elections. A rising etcd leader-change rate is an early warning of disk or network trouble. We graph it and alert on it.
Benchmark storage first. Before etcd goes on any machine, we run fio and check the latency profile. We test the disk before we trust it.
Keep etcd's data small. A smaller, healthier database means faster maintenance. We make sure history compaction runs, we defrag on a schedule, and we keep an eye on object count and churn. High-churn objects like events can be moved to a separate etcd to protect the main one.
Write a runbook. We wrote down the exact chain and the exact commands to check fsync latency, leader elections, and db size. Next time, the on-call engineer starts at the disk, not at the apps.
The lesson
The real lesson is about where to look. The symptom was at the top of the stack — random pods dying. The cause was at the very bottom - a disk that could not fsync fast enough. Everything in between, etcd, Raft, leases, the API server, was just faithfully passing the slowness up the chain.
In a distributed system, "is this thing alive?" almost always means "can it renew a small claim before a deadline?" And every one of those claims, in k8s, is a write that must reach etcd's disk. So when pods start dying for no reason and the timing looks oddly regular, do not stare at the data plane, and start at the disk under etcd first.
