I've come across many a blog posts describing ETCD restoration process on a single node, but not on a full 3 master setup which is what everyone in production uses at the very least. This post is the failure catalog we wish we had read first.

Firstly: Identify the 'Failure Mode'

A restore rewinds the whole cluster to the snapshot instant, discarding every write since. That is fine when you have lost everything, but it might well be an unnecessary data loss you've inflicted on yourself in other cases.

A 3-member etcd cluster needs a majority, 2 of 3, to serve writes. So triage before you touch anything as getting it right is half the value:

• 1 master down. Quorum retained (2/3), cluster fully operational - Remove the dead member, add a fresh one.

• 2 masters down. Quorum lost (1/3) - read-only, but the survivor still holds current data. Do NOT restore. Recover quorum on the survivor with --force-new-cluster, then re-add members.

• All 3 down. No surviving data. This, and only this, is a snapshot restore.

The trap

The obvious idea for all-3-down is, "restore the snapshot on each master". It is wrong, and it fails in a way that wastes an hour of your time eating into your precious RTO. etcdutl snapshot restore mints a brand new cluster ID every time it runs - Restore on all three and you get three separate one-member clusters, three different cluster IDs, that will never form a quorum together.

The symptom is master1 endlessly probing the other two as peers, "connection refused" or a cluster ID mismatch in the logs, and no leader, ever.

Correct shape: restore once, then add

The procedure that actually works, and the one our DR plan mandates, is:

1. Restore the snapshot exactly once, on master1, as a single-member cluster.

2. Add master2, then master3, as new members with etcdctl member add. They start with an empty data dir and replicate from master1.

One restore followed by two live joins. Concretely, once master1 is up as a single healthy member, each joiner is a register-then-start, run from master1:

# 1. register the joiner in the cluster (from master1)
etcdctl member add master2-<cluster> --peer-urls=https://10.0.0.2:2380 \
  --endpoints=https://10.0.0.1:2379 $CERTS
# 2. on master2: set --initial-cluster-state=existing and --initial-cluster to the
#    growing member list in its etcd manifest, empty /var/lib/etcd, then start etcd
# 3. confirm from master1 that master2 is 'started' before you touch master3
etcdctl member list -w table --endpoints=https://10.0.0.1:2379 $CERTS

We wrapped the whole sequence in an Ansible role, etcd-restore, so the real run is one command:

ansible-playbook -i inventory/<cluster> etcd-restore.yaml \
  -u <user> -kK -b -e target_cluster=<cluster> -e etcd_snapshot_db=/root/etcd-snapshot.db

However the manual steps still matter, because break-glass is real, and because every one of the following issues is something the automation had to learn to survive.

etcdutl snapshot restore /root/etcd-snapshot.db \
  --name master1-<cluster> \
  --initial-cluster master1-<cluster>=https://10.0.0.1:2380 \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd/.restore-tmp
mv /var/lib/etcd/.restore-tmp/member /var/lib/etcd/member
rmdir /var/lib/etcd/.restore-tmp

On etcd 3.6, etcdctl snapshot restore was removed, so the offline restore must use etcdutl.

etcdctl is still what you use for the online member add and endpoint health later. You need both binaries, matched to the cluster's etcd version.

Landmine 1: the member name must match the manifest

The --name and the --initial-cluster key you restore with have to exactly equal the --name in master1's etcd static Pod manifest, which is the node's full hostname, not the literal "default" that most examples use. Restore with the wrong name and the member never lines up with its peer URL mapping, and the cluster will not form. Read the real value straight from the manifest, do not guess.

Landmine 2: --initial-cluster-state=existing must actually be present

When you add master2 and master3, each joiner's manifest needs --initial-cluster-state=existing plus an --initial-cluster listing every member up to and including itself. Miss the state=existing line and etcd defaults to new, so the joiner bootstraps its own cluster instead of joining, and you are back to the cluster ID mismatch from the trap above.

Landmine 3: you cannot add a member to an unhealthy cluster

etcd refuses member add if the current cluster is not healthy, with "etcdserver: unhealthy cluster". After you add master2, it has to replicate the full keyspace before it is a healthy voter, and that can take tens of seconds for a large DB. Add master3 during that window and the request is rejected, because going from 2 voters to 3 while only 1 is healthy would break quorum. So you gate each join on the current cluster being fully healthy first, and you add members strictly one at a time.

The same rule bites during any restart. Never bounce an etcd member unless the other two are healthy, or you drop below quorum. We learned this the hard way when an optional "tidy the manifests" step restarted a member while another was still down, and froze the cluster.

Landmine 4: Do not boot stale data

If master1's old data dir survives, because a wipe was skipped or incomplete, the move-into-place step can be silently skipped and etcd boots the previous cluster's multi-member data, then hangs probing peers that no longer exist. Guarantee the data dir is empty before you restore, do not rely on a "create only if missing" guard to protect you.

The revision-bump problem

There is a subtle failure that survives everything above, and only the official etcd docs really mention it. It is about revisions.

Every write to etcd bumps a global revision counter, and k8s resourceVersion is derived off of it. Controllers and operators do not poll, they use informers, which are watch-based local caches that track "I have seen up to RV X" and get notified on changes.

Here is the problem - a restore rewinds etcd's revision to the snapshot's value, which is lower than the RVs clients were already holding before the disaster. New writes then count up from that lower number, so fresh changes get RVs below what informers believe they have already seen. The watch caches do not refresh correctly, and controllers quietly act on stale state or miss post-restore changes entirely.

etcd added two etcdutl snapshot restore flags specifically for this, and they must be used together:

• --bump-revision <n>, add n to the restored keyspace's revision so revisions never move backward.

• --mark-compacted, mark that bumped revision as the compaction point, which terminates existing watches and makes etcd refuse to serve revisions after the snapshot, forcing every informer to re-list cleanly. It is required when --bump-revision > 0, and disallowed otherwise.

A common sizing, straight from the etcd docs, is to bump by 1,000,000,000, which covers a week-old snapshot at under 1500 writes per second.

etcdutl snapshot restore /root/etcd-snapshot.db \
  --name master1-<cluster> \
  --initial-cluster master1-<cluster>=https://10.0.0.1:2380 \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd/.restore-tmp \
  --bump-revision 1000000000 --mark-compacted

What it looks like when you skip it. This is the part the docs do not cover. The cluster comes up, kubectl get returns correct data, so it looks healthy. But watch-driven components misbehave:

• Controllers and operators do not react to changes made after the restore. You edit a Deployment or a custom resource (CR) and nothing reconciles, no new ReplicaSet, no operator action.

• Component logs fill with "too old resource version", "the server has received a request that reused a resource version", or repeated re-list loops.

• Leader election, which rides on RVs in Lease objects, can flap.

How to diagnose it. The tell is that reads work but watches do not. So:

• Check etcd's current revision, etcdctl endpoint status -w json, and notice it is far below where the cluster was before the restore.

• Grep kube-controller-manager, kube-scheduler, and operator logs for "too old resource version" and re-list churn.

• Drop a marker, create a ConfigMap or bump an annotation, and watch whether its consumer notices. If a known-good reconcile does not fire while kubectl get shows the object fine, your informers are stale.

If you already restored without the flags: the only practical fix is to force every watch to re-list, restart kube-controller-manager, kube-scheduler, and any operators, and restart kubelets for the data plane. Doing the restore with --bump-revision --mark-compacted in the first place is far cleaner, because it invalidates the caches for you.

Think of it as the CP counterpart to the kubelet restart below. bump-revision fixes informer staleness for controllers and operators, kubelet restart fixes it for the node agents. You want both.

After the restore, reconcile the data plane

A restore rewinds the API state, but not the workloads actually running on nodes, and every kubelet's cached watch is now stale because etcd's revision jumped backward. So kubelets log errors and Pods drift from what etcd believes. In my tests the fix is a rolling systemctl restart kubelet across every node, CP and workers, one at a time, each gated on the node returning Ready, and for masters gated on etcd staying healthy between restarts. A kubelet restart does not kill running containers, it just forces kubelet to re-list, re-watch, and reconcile.

Confirm you actually recovered

Do not declare victory on "the Pods look up". Check the things that prove the cluster is genuinely whole:

• etcdctl endpoint status -w table and endpoint health across all three endpoints, expect three members, one leader, all healthy, IS LEARNER=false.

• kubectl get nodes, every node Ready.

• The marker test from the revision section, a fresh object actually gets reconciled, which proves watches are live, not just reads.

That last check is the one people skip, and it is the one that catches the revision-bump problem before your users do.

TLDR: The Summary

• Triage first. Only all-3-down is a restore.

• Restore once on master1, then member add master2 and master3, one at a time, each gated on cluster health.

• Respect the mount, match the member name, force state=existing, never bounce a member without quorum, and never boot stale data.

• Restore with --bump-revision --mark-compacted so controllers' informers do not go stale on the rewound revision.

• Restart kubelet fleet-wide afterward to resync the data plane.

• Confirm with member health plus a live reconcile test, not just kubectl get.

• Time it end to end, snapshot fetch included.

None of the individual pieces are exotic. The value is in the order, the gates, and the specific ways it breaks, which is the stuff you do not want to be discovering live at 3am with an outage bridge listening.

Sources

• Operating etcd clusters for Kubernetes, kubernetes.io

• Disaster recovery, etcd.io

• Restoring etcd quorum, Red Hat OpenShift docs

• Restoring a kubeadm Kubernetes Cluster from an etcd Backup, Medium

• ocp42-etcd-backup-restore-ansible, GitHub

• Restoring etcd without breaking API guarantees (aka etcd restore for Kubernetes), etcd issue #16028

• etcdutl README (snapshot restore flags), etcd-io/etcd

• the backup has to be boring and reliable, and

• the retrieval has to work even when nothing else does.

That second half is where most write-ups I've come across quietly wave their hands. They show you kubectl cp from a backup Pod, which is lovely right up until the day you actually need it, when the API server (API) is down and kubectl returns nothing but connection errors.

This post is Part 1, how we take snapshots and how we get one back with the cluster fully dead. Part 2 covers the restore itself, which is where the real landmines are.

The backup

A k8s CronJob in a dedicated namespace runs every 1h and calls etcdctl snapshot save against the live cluster. This gives us a recovery point objective (RPO) of roughly 1h. That is a deliberate tradeoff based on our internal calculations; a tighter RPO means more frequent snapshots and more storage churn, and you should size it to how much state your cluster can afford to lose.

The snapshot lands on a NetApp TRIDENT NFS-backed PVC with a sidecar pruning anything older than 30 days.

• The PVC itself is normally reached only through a small validation Deployment that we keep scaled to 0 from within the cluster

• Or NFS mountable on one of the master nodes for a more direct and traditional access.

Two facts about that .db file drives everything downstream:

1. It is a full, point-in-time (PIT) copy of the entire keyspace. Restoring it rewinds the whole cluster to that instant.

2. It contains every Secret in plaintext. etcd at rest is usually unencrypted, so the snapshot is exactly as sensitive as the cluster it came from. Treat every copy as mode 0600 and delete it the moment you are done.

Retrieval: part that actually matters

Getting the snapshot back has two very different modes, and the gap between them is the whole point of this post.

Mode A, the cluster is still up. This is the HAPPY path.

• Scale the validation Pod to 1

• kubectl exec in, pick the newest .db

• kubectl cp it out,

• scale back to 0.

Mode B, the cluster is down. The SAD path, because Mode A depends on a working API and kubectl, which is precisely what a real DR event takes away. The good news though is that the backup lives on an NFS export, and we skip k8s entirely and mount the export directly on a surviving master, which still has L3 reach to the storage data LIF.

Mode C, offsite object storage. We also ship every snapshot to blob storage, which is the cleanest retrieval when it works, one authenticated pull with a CLI

<cloud> storage blob download --container etcd-backups \
  --name etcd-snapshot-<...>.db --file /root/etcd-snapshot.db

The catch is credentials**. In our outage the blob access keys themselves were "lost" due to the extent of the outage, which is exactly why we fell back to the NFS mount in Mode B.**

The blob failure taught a real lesson - one retrieval path at times may not be enough. So we promoted the NFS mount from a manual break-glass step to a first-class, automated fetch mode that sits alongside the simple blob download - two independent paths.

Mounting the backup export on a master

You need two coordinates, the NFS server address and the export path. Record them now, while the cluster is healthy, because you cannot ask the PV for them once the API is gone:

PV=$(kubectl -n ckp-cluster-jobs get pvc ckp-cluster-jobs-etcd-backup-data -o jsonpath='{.spec.volumeName}')
kubectl get pv "$PV" -o jsonpath='{.spec.nfs.server}{" "}{.spec.nfs.path}{"\n"}'

Stash those in the Ansible per-cluster inventory (group_vars) so they are already on disk before you need them. During DR, the manual form is just a read-only mount, copy the newest file off, unmount:

mkdir -p /mnt/etcd-backup
mount -t nfs -o ro,nfsvers=4.1 <server>:<export> /mnt/etcd-backup
ls -ltr /mnt/etcd-backup
cp /mnt/etcd-backup/etcd-snapshot-<...>.db /root/etcd-snapshot.db
umount /mnt/etcd-backup

Read-only is a safety guardrail - the mount physically cannot modify or delete your backups, which is exactly the guarantee you want when you are already having the worst day of your quarter.

Verify before you trust

A snapshot you have not checked is a guess. Before you build a restore around it, confirm it is intact and confirm it is the point in time you think it is:

etcdutl snapshot status /root/etcd-snapshot.db -w table

That prints the hash, total keys, and the revision. The revision matters, it tells you roughly how recent the snapshot is, and it is what a PIT restore lands on.

If this command errors, the file is truncated or corrupt, and you want to know that now, not halfway through a restore.

Automation: the etcd-snapshot-fetch role

Doing the mount dance by hand at 3am invites mistakes, so we wrapped it in an Ansible role, etcd-snapshot-fetch, driven by a playbook that targets one master (master1 by default):

ansible-playbook -i inventory/<cluster> etcd-snapshot-fetch.yaml \
  -u <user> -kK -b -e target_cluster=<cluster>

The flow is:

1. Assert the target host actually belongs to the named cluster becasue the last thing we'd want in this scenario is to have 2 outages instead of one :-)

2. Mount the export read-only on the master.

3. Find the newest etcd-snapshot-*.db, or a specific one you name for a PIT restore.

4. Copy it off the read-only mount to local disk on the master, then fetch it to the Ansible control host.

5. Always unmount and clean up, in an always: block, so the export is never left mounted even if a step fails.

The output is a verified snapshot on your control host at 0600, ready to hand to the restore. Notice the security posture end to end.

What Part 1 buys you

You now have a verified snapshot in hand, retrieved through a path that does not depend on the cluster being alive. That independence is the entire point. A backup you can only reach through the thing that just died is not a backup, it is a hope.

There is one more habit worth building - rehearse it. A backup nobody has ever restored is Schroedinger's backup, simultaneously fine and useless until you open the box, which is a good segue into Part 2 as that is where we open the box, and where restoring onto a 3-master cluster goes wrong in ways that surprised us.

Sources

• Operating etcd clusters for Kubernetes, kubernetes.io

• Disaster recovery, etcd.io

• Backing up etcd data, OKD docs

• From Disaster to Recovery, a Practical Case Study on Kubernetes etcd Backups, dev.to

Surprisingly, I realized recently that I have a much better grasp on containers than vms. If you ask most platform engineers or devops folks, they can talk confidently about namespaces, cgroups, and container runtimes, but when it comes to what a vm or a microvm is actually doing down at the silicon level, things get a bit fuzzy.

Here is my breakdown of how vms actually work underneath, why they can feel slow, and how modern microvms fix those bottlenecks.

The Privilege Problem: Rings of Trust

To understand vms, we have to look at how cpus handle security. Processors use privilege levels called "rings" to protect the system.

• Ring 0 is the most privileged mode—reserved for the os kernel so it can interact directly with physical h/w, memory, and devices.

• Ring 3 is restricted, meant for user space apps.

In a normal bare-metal setup, your host os owns Ring 0. But when you introduce a traditional vm, the guest os inside that vm also naturally expects to run in Ring 0.

A bit of throwback first - before h/w virtualization existed, this caused a major conflict. The hypervisor had to force the guest os down into Ring 1 or Ring 3. Every time the guest os tried to execute a privileged kernel command, the cpu would throw an exception. The hypervisor had to trap that error, translate the instruction on the fly, and emulate the response. This "trap-and-emulate" cycle is exactly why old-school vms felt brutally slow.

Enter Hardware Virtualization (VT-x and AMD-V)

To fix this performance bottleneck, cpu manufacturers added specialized hardware extensions (Intel VT-x and AMD-V). They introduced a completely new execution dimension: Root Mode and Non-Root Mode.

• Root Mode: Where your host os and hypervisor (like KVM) live.

• Non-Root Mode: A separate sandbox containerized by the cpu hardware for the guest vm.

Inside Non-Root Mode, the guest os gets its own complete set of rings, including its own Ring 0. It thinks it has absolute control over the machine.

This splits the lifecycle of an instruction inside a vm into two distinct paths:

1. The Fast Path (99% of work)

When an app inside the vm does basic math, runs loops, or processes data, it fires standard instructions. The physical cpu checks the state, sees it's safe, and executes it directly on the physical silicon at bare-metal speed. The hypervisor isn't involved at all.

2. The Hypervisor Path (The 1%)

When the app needs to write a file to disk or send a packet over the network, the guest os fires a privileged hardware instruction. The physical cpu flags this, pauses the vm, and executes a hardware-level context switch called a vm-exit.

Control is handed back to KVM in Root Mode to safely perform the real physical operation. Once done, it triggers a vm-entry to unpause the vm and hand control back to the guest. Because this context switching is baked directly into the cpu silicon, it happens in nanoseconds, keeping the overhead minimal.

The Catch: Why Traditional VMs Still Feel Heavy

Even with h/w virtualization, standard vms take time to boot and consume significant resources. This isn't because of instruction execution; it's because of hardware emulation bloat.

Traditional hypervisors (like QEMU) emulate decades of legacy hardware to support any random os you might install. They emulate virtual IDE controllers, floppy drives, PCI buses, and ancient graphics cards. Booting a traditional vm means waiting for a full, bloated guest kernel to initialize all these virtual devices.

Shifting to MicroVMs

If you are running modern workloads—like serverless functions or untrusted AI agent code—you need the absolute isolation of a vm kernel boundary, but you can't afford a 10-second boot time or hundreds of megabytes of baseline memory overhead.

This is where microvms (like Firecracker) come into play. A microvm strips away almost all legacy emulation. It throws out the floppy drives and video cards, providing a hyper-minimal device model (usually just virtio for basic block storage and network interfaces).

Because the guest kernel is stripped down to the bare essentials, a microvm can boot in less than 200 milliseconds and uses a fraction of the ram of a standard vm.

The Under-the-Hood Stack

What makes a microvm fascinating is that it combines both hardware virtualization and traditional container primitives:

• Hardware Boundary: The code inside the microvm is secured via KVM using the hardware's non-root mode, ensuring a compromised app or agent cannot escape to the host kernel.

• Host Boundary: On the host side, the Virtual Machine Manager (VMM) process running the microvm is wrapped in a "jailer" that uses standard Linux namespaces (ns), control groups (cgroups), and seccomp filters.

This gives you a defense-in-depth architecture. Even if an untrusted workload manages to break the guest kernel, it's still trapped by the hypervisor. And if it somehow breaks the hypervisor process, it hits the cgroups and namespaces boundary on the host node.

Next up would be a detailed breakdown of 2 of the sandboxes that I intend to trial for securing ai agent related workloads in ther cloud - just-bash vs microsandbox!

Stay tuned!

The strange part: running pods do not need etcd

Before the cause, one fact that our team (half, atleast) was well aware is worth saying first - a pod that is already running does not talk to etcd. It does not even talk to the API server to keep running. Once the kubelet has started a container, the app just runs and serves traffic. You can take the whole control plane offline and your running pods will keep working.

This knowledge precisely made things a lot harder to debug.

The cause: a slow disk breaks the heartbeat of the cluster

etcd is the database that holds all k8s state. It is built for strong consistency, and it pays for that with one strict rule: every write must be saved to disk before it counts. Technically, etcd writes to a write-ahead log (WAL) and calls fsync to force the bytes onto the disk. Only after that fsync returns does the write commit.

etcd also runs as a small cluster of members that agree using the Raft algorithm. Every write goes through the leader and must be confirmed by a quorum, and each member must fsync it. This means one thing that becomes very important later: if the leader's disk is slow, every write in the whole cluster slows down at once.

Now connect this to pods. Many parts of k8s stay alive by renewing a "lease" before a deadline. A lease is just a small record that says "I am still here," and renewing it is a write - which means a write to etcd, which means an fsync. There are two leases that matter here:

• Leader election leases. Control-plane components like the scheduler, the controller-manager, and many operators run more than one copy, but only one is active at a time. The active one must renew its leader lease every few seconds. If it cannot renew in time, it can no longer prove it is the leader. These programs are written to do the safe thing in that case: they exit on purpose, so that two leaders never act at once.

• Node leases. Each node's kubelet renews a node lease about every 10 seconds to say "this node is alive." If those renewals stop arriving, the control plane marks the node NotReady, and after a grace period it starts evicting the pods on it.

Here is the full chain in one line:

disk I/O stall → fsync is slow → etcd WAL commit is slow → Raft writes back up → leases cannot renew in time → leader components exit and nodes go NotReady → pods restart or get evicted.

That is why our pods "died." A leader-elected pod killed itself because it could not renew its lease in time. An ordinary pod was evicted because its node missed too many heartbeats. In both cases, the thing that failed was a small write that could not fsync through a disk-stalled etcd. Our disk had short latency spikes every few minutes, and those spikes were printing themselves onto the pod crash schedule.

One more detail made it worse. The API server's own health check includes an etcd check. When etcd was slow, the API server itself started returning errors. So a single slow disk did not fail gently - it knocked over the one dependency that every lease and heartbeat runs through, all at the same time.

The debugging: looking in the wrong place first

The hard part of this kind of incident is that the symptom and the cause are far apart. We lost a few hours looking at the wrong layers. Here is the path we took, including the dead ends, because the dead ends are useful.

First we blamed the apps. The pods that died had no pattern at the app level. Different images, different teams, different ns. That told us it was not an app bug. If it were, it would hit one app, not all of them.

Then we blamed networking and resources. We checked CPU and memory limits, OOM kills, and network policies. Nothing lined up. The crashes did not match any resource graph.

Then we saw etcd's metrics rising - and talked ourselves out of it. Somewhere in here we did glance at etcd. The fsync and WAL commit times were climbing, But we waved it off. etcd sits far below the apps in the stack, and the pods that were dying were many links away from it. so we filed the etcd numbers under "probably related but unrelated" and kept looking elsewhere.

Then we noticed the timing. The crashes came in waves, every five to ten minutes. Random pods, but a regular beat. Random-but-periodic is a strong clue. It means something shared and low-level is failing on a cycle, not one app misbehaving.

Then we read the control-plane logs. This is where it turned. The API server logs showed timeouts talking to etcd. The etcd logs showed leader elections happening again and again. A healthy etcd almost never changes leader. Frequent leader elections almost always mean the disk or the network is too slow.

Then we went back to the etcd metrics we had dismissed. Once the logs pointed at etcd, we stopped waving off those numbers and actually read them. Two metrics tell the story directly:

etcd_disk_wal_fsync_duration_seconds etcd_disk_backend_commit_duration_seconds

On a healthy cluster the p99 of these should be a few milliseconds. Ours were spiking past 100ms, again on that same five-to-ten-minute beat. A good rule of thumb: if p99 of the WAL fsync is consistently above 100ms, you have a storage problem, not a k8s problem. These were the same numbers we had seen an hour earlier and ignored — only this time we let ourselves believe them.

Then we confirmed it at the disk as the metrics showed high latency and a disk pinned near 100% busy, in the same five-to-ten-minute waves. We could also see other workloads driving I/O on that same device at those exact moments. That was the final answer: etcd was sharing a disk with noisy I/O, and that disk could not keep fsync fast and steady.

The resolution: give etcd a disk it does not have to share

The fix was not in k8s settings. It was in storage.

The root problem was that etcd's disk was shared with other heavy I/O. So our first and most important change was simple: give etcd its own dedicated, fast, low-latency disk - a local NVMe SSD - and keep all other heavy I/O off it. No container storage, no logs, no batch jobs on the same device.

There are storage tricks that make fsync return instantly by not really waiting for the write to land (for example, disabling sync writes at the filesystem). These make the symptom disappear, but they break etcd's one job: durability.

With a rolling upgrade we moved the entire etcd cluster to dedicated NVMe, and immeidately witnessed the fsync p99 dropped back to a few milliseconds, the leader elections stopped, and the random pod crashes stopped with them.

The action: so it never surprises us again

Fixing the disk ended the fire. These changes make sure it does not start again, and that if it does, we see it early.

• Alert on the real metric. We now alert when p99 of etcd_disk_wal_fsync_duration_seconds goes above a safe threshold. This catches the problem while etcd is still slow, before it becomes pod chaos.

• Watch leader elections. A rising etcd leader-change rate is an early warning of disk or network trouble. We graph it and alert on it.

• Benchmark storage first. Before etcd goes on any machine, we run fio and check the latency profile. We test the disk before we trust it.

• Keep etcd's data small. A smaller, healthier database means faster maintenance. We make sure history compaction runs, we defrag on a schedule, and we keep an eye on object count and churn. High-churn objects like events can be moved to a separate etcd to protect the main one.

• Write a runbook. We wrote down the exact chain and the exact commands to check fsync latency, leader elections, and db size. Next time, the on-call engineer starts at the disk, not at the apps.

The lesson

The real lesson is about where to look. The symptom was at the top of the stack — random pods dying. The cause was at the very bottom - a disk that could not fsync fast enough. Everything in between, etcd, Raft, leases, the API server, was just faithfully passing the slowness up the chain.

In a distributed system, "is this thing alive?" almost always means "can it renew a small claim before a deadline?" And every one of those claims, in k8s, is a write that must reach etcd's disk. So when pods start dying for no reason and the timing looks oddly regular, do not stare at the data plane, and start at the disk under etcd first.

As a senior engineer, you get used to things breaking. But sometimes, the root cause is hiding a few layers deep in the infrastructure. Recently, our company messenger app went down completely. I was looped into the incident bridge to figure it out.

Here is a quick write-up of the investigation, how we traced it from K8s all the way down to a container runtime default, and what we learned.

The Outage: Starting at the Top

Our app stack is mostly running on a K8s platform. Naturally, that is the first place I looked when the alerts started firing.

I opened the logs for the main messenger workload. Immediately, I saw that the application was throwing 503 Service Unavailable errors when trying to call one of its downstream services.

In our architecture, the communication to this downstream service goes through an HAProxy load balancer layer. Some people always want to blame the load balancer first. To verify, we checked the HAProxy logs.

HAProxy was showing connectivity issues to the backend with 504 Gateway Timeout. This actually ruled HAProxy out. It was doing its job, trying to pass traffic, but the backend was not responding in time.

Moving to the Downstream Service

Next, we checked the downstream backend service in K8s. The pods for this service were in a CrashLoopBackOff state.

We looked at the logs right before the crash. We figured out that the service was trying to talk to our Cassandra database cluster, but the response latency had increased massively. The BE app. was basically timing out waiting for the database, exhausting its connections, and crashing.

So, the backend was just a victim. The real problem was in the database layer.

The Database Layer: The Real Culprit

We moved our investigation to the Cassandra nodes. For this setup, Cassandra is not running inside K8s. It is running on instances using Podman containers.

When we checked the cluster status, we saw that 3 to 5 nodes had been OOM'ing (Out of Memory) since the morning. They were constantly getting killed and restarted.

We went into Grafana and pulled the Loki logs for the Cassandra containers. We found this exact error spamming the logs:

2026-06-21 17:44:44.408 java.lang.OutOfMemoryError: unable to create new native thread

The "Aha!" Moment

When you see an `OOM Error` in Java, the first thought is usually heap exhaustion. But the unable to create new native thread part is very specific.

This error doesn't mean the JVM ran out of RAM. It means the operating system (or the container runtime) is refusing to let the JVM spawn any more threads. Cassandra is a very heavy Java application; it relies on creating a lot of native threads for connection pooling, compaction, and request handling.

We investigated the host and the container configs. That is when we found it: the Podman default thread limit (pids-limit).

By default, Podman sets a limit on the number of processes/threads a container can create to prevent fork bombs. During high traffic that morning, Cassandra tried to scale up its thread count to handle the load. It hit the Podman default PIDs limit limit. Because the JVM was blocked from creating the threads it needed to operate, it panicked, threw the OOM error, and crashed the node.

This caused a chain reaction:

1. Cassandra nodes hit thread limit and crash.

2. Database latency spikes for surviving nodes.

3. Downstream K8s service times out waiting for DB, then crashloops.

4. HAProxy throws 504s.

5. Frontend app gets 503s and goes down.

The Immediate Fix

To resolve this, we had to update the container configuration for the Cassandra nodes to increase or remove the pids-limit restriction, allowing the JVM to spawn the threads it actually needed for our production scale. Once we restarted the Podman containers with the new limits, the database stabilized, the downstream service recovered, and the messenger app came back online.

Bigger Picture: Architectural Changes

Fixing the Podman thread limit was the immediate band-aid to stop the bleeding. But incident reviews are about finding long-term structural fixes.

Our single massive production database had too large of a blast radius. The biggest takeaway from this outage was the decision to split our single production instance into two separate environments: Large Prod and Small Prod, each tailored to different audienses..

1. Moving Large Prod to k8s: For our main, larger prod. workload, we are migrating Cassandra into our k8s platform. Yes, I know running heavy stateful databases in K8s is a controversial topic. But for us, having our database on the same control plane as our application stack brings operational consistency. It gives us better automated distribution, easier pod anti-affinity rules, and faster recovery times when a node dies, compared to managing standalone container instances.

2. Moving Small Prod to Bare Metal For the smaller production cluster, we are going the opposite direction: pure bare metal. Yes, maintaining bare metal adds a different layer of operational complexity because we have to manage the physical nodes directly using our automation playbooks instead of K8s manifests. However, removing the container runtime, the virtualized networking layer, and the kernel isolation overhead gives us the absolute maximum raw disk I/O and compute performance for our most latency-sensitive workloads.

Sometimes, the best architectural decision is knowing when to use the shiny orchestration tool, and when to just let a database talk directly to the hardware.

Takeaway

When you are managing infrastructure, you monitor CPU and memory limits very closely. But resource limits like process/thread counts (PIDs) are easy to forget until they break production. Always check the default limits of your container runtime (Docker, containerd, Podman) because their defaults are usually meant for generic lightweight web apps, not heavy enterprise databases.

We tune resource requests, optimize autoscaling, benchmark storage backends, and chase n/w bottlenecks. Yet over the course of managing various platforms over the years I've noticed that the most impactful performance wins rarely come from simply adding more CPU, mem. or storage - they come from reducing the amount of work the OS has to do in the first place.

Few days back chasing a production bug sent me down an interesting rabbit hole. It started with investigation that lead to a Linux syscall called sendfile, took me through zero-copy networking and DMA, and eventually helped me understand why eBPF, XDP, and Cilium have become such important building blocks for modern k8s platforms.

These technologies solve different problems, but they share one philosophy:

The fastest operation is the one you never have to perform.

A Pattern Hidden Throughout Linux

When we hit a performance issue, our instinct is to ask:

How can I make this operation faster?

Kernel developers tend to ask something different:

Can I avoid doing this operation altogether?

That mindset shows up everywhere in the kernel. Instead of copying data, traversing networking layers, context switching between kernel and userspace, or eval'ing thousands of rules per packet, Linux keeps evolving toward doing less.

Once you see the pattern, technologies like sendfile(), eBPF, and Cilium stop looking like unrelated tools and start looking like the same idea applied in different places.

The Cost of Moving Data

Consider a traditional app serving a file over the network. A normal implementation that I've comre across most generally -

read(file_fd, buffer, size);
write(socket_fd, buffer, size);

The data path looks roughly like:

Disk
 ↓  (DMA copy)
Kernel Page Cache
 ↓  (CPU copy)
Userspace Buffer
 ↓  (CPU copy)
Kernel Socket Buffer
 ↓  (DMA copy)
NIC

That's 4 copies of the same bytes before they leave the machine — two of them burning CPU and polluting its caches - plus 4 context switches, since read() and write() each cross into the kernel and back.

Each copy costs CPU cycles, memory bandwidth, and cache capacity. At small scale it's negligible. At large scale it's expensive.

How sendfile() Changes the Equation

Linux added sendfile() to cut out the copies that route through userspace. Instead of read() then write(), the app calls:

sendfile(socket_fd, file_fd, NULL, size);

In the best case the path collapses to:

Disk
 ↓  (DMA copy)
Kernel Page Cache
 ↓  (DMA gather copy → refs., no cpu copy)
NIC

The app never touches the payload, and we drop from four context switches to two.

One honest caveat worth keeping: this true zero-copy path only happens when the NIC supports scatter-gather DMA and checksum offload. There, the kernel hands the NIC references to the page-cache pages and the hardware gathers them directly. Without that hardware support, sendfile() still saves you the two userspace copies, but there's one remaining kernel-side copy from the page cache into the socket buffer. So sendfile() is "fewer copies" always, and "zero-copy" when the hardware cooperates.

Either way, the lesson isn't the syscall. It's the design principle behind it:

Avoid moving data across boundaries unless the data has changed.

DMA: The Foundation Underneath

That zero-copy path doesn't work by magic. It leans on something much older than sendfile() — DMA (Direct Memory Access).

DMA predates all of this by decades, and it's precisely why the page-cache-to-NIC hop above can skip the CPU. Modern hardware can move data directly between devices and memory without the CPU touching every byte:

NVMe SSD ↔ RAM
NIC      ↔ RAM
GPU      ↔ RAM

The CPU's job shrinks to:

1. Configure the transfer

2. Initiate it

3. Get out of the way

sendfile() is really just the kernel arranging for DMA to do the heavy lifting. Same theme, one layer down:

Reduce unnecessary CPU involvement.

The Same Problem Appears in k8s Networking

Take a packet entering a cluster that uses iptables-based svc routing. A simplified path:

NIC
 ↓
Kernel Networking Stack
 ↓
Conntrack
 ↓
iptables Rules (NAT + forwarding)
 ↓
Pod

Every layer does work. Every lookup costs cycles. Every packet traverses multiple subsystems before it reaches its destination.

At moderate scale this is fine. At large scale — service meshes, API gateways, observability pipelines, high-throughput microservices, AI inference workloads — the networking stack itself becomes a meaningful CPU consumer.

The Traditional kube-proxy Challenge

Historically, k8s svc load balancing leaned heavily on iptables. iptables is wonderfully flexible, but it wasn't designed for cloud-native environments slinging massive east-west traffic at scale.

The packet inspection work is done by the kernel's netfilter. And The real problem is how those rules are evaluated. In iptables mode, rules are a sequential list, so match time scales roughly linearly (O(n)) with the number of svc and endpoints. For every packet the kernel may walk through:

Packet
 ↓
Rule Matching (linear scan)
 ↓
NAT Processing
 ↓
Conntrack Lookup
 ↓
Forwarding Decision

As svc count grows, so does the chain you scan per packet. This is exactly why kube-proxy later gained an IPVS mode, which uses hash-table lookups (closer to O(1)) instead of a linear scan — a meaningful improvement that predates eBPF entirely.

So to conclude iptables was traditionally just not designed for such large volume of work. It's that, for this workload, the system was doing more work than necessary.

eBPF: Bringing Logic Closer to the Data

This is where eBPF gets interesting.

Worth saying up front: eBPF is much bigger than networking. It powers tracing, observability, and security (seccomp-BPF, LSM hooks, and more). But in this story, the relevant trick is networking.

eBPF lets you run sandboxed programs directly inside the kernel. Instead of pushing packets through layer after layer, your forwarding logic can run much earlier in the packet's journey.

Traditional path:

Packet
 ↓
Networking Stack
 ↓
iptables (linear scan)
 ↓
Additional Processing
 ↓
Destination

eBPF path:

Packet
 ↓
eBPF Program
 ↓
Destination

The payload is identical. What's reduced is the work needed to make a forwarding decision.

Why Cilium (and other eBPF based solutions) Matters

When I first started working with Cilium, I realized it embodies the same principles that motivated sendfile().

Cilium uses eBPF to move networking decisions closer to where packets enter the kernel. Instead of relying primarily on large iptables rule sets, svc handling can run as efficient kernel-resident eBPF programs. (In full kube-proxy replacement mode it removes the iptables-based svc path entirely; some configurations still keep iptables around for parts of the stack.)

The result is often:

• Lower latency

• Reduced CPU consumption

• Faster svc lookups (hash/map-based instead of linear)

• Better scalability at large cluster sizes

The goal isn't to make packet processing faster. It's to do less packet processing.

A Common Thread Across Linux Innovation

Step back and a lot of Linux performance work follows one pattern.

sendfile()

Avoid unnecessary data copies.

DMA

Avoid unnecessary CPU involvement. (And it's the foundation sendfile() stands on.)

io_uring

Avoid unnecessary syscall overhead, via shared submission/completion rings instead of a syscall per op.

eBPF

Avoid unnecessary networking-stack traversal.

XDP

Avoid unnecessary processing by running at the driver level, before the kernel even allocates an sk_buff.

Cilium

Avoid unnecessary svc-routing overhead inside k8s.

Different subsystems. Same philosophy.

What This Changed for Me as a k8s Engineer

Studying sendfile(), DMA, eBPF, and Cilium changed how I think about performance. Now when I chase a bottleneck, I ask:

How many times is the data being copied?

How many components are touching this packet?

How many layers are involved in this request?

Can some of this work be eliminated entirely?

Those questions surface opportunities that CPU graphs and memory dashboards never might.

Final Thoughts

The most valuable thing I took from this wasn't a syscall or a CNI. It was a recurring pattern in Linux design: the kernel community keeps finding ways to eliminate work.

Less copying. Less context switching. Less packet processing. Less CPU involvement.

Whether it's sendfile(), DMA, io_uring, eBPF, XDP, the principle holds:

The best optimization is often removing work rather than accelerating it.

As k8s engineers, that mindset is worth carrying into our own platforms. Next time you're debugging a performance issue, don't only ask:

How can I make this faster?

Ask:

What work can I avoid doing altogether?

An honest admission first up - we had no real prior experience running workloads in the cloud, so it was a genuine step into the unknown, and it was going to be hard. But the mindset going in was that we had to make it work, and that whatever we learned along the way would outlast the immediate deadline, and that turned out to be true.

I'll say upfront that the architecture that follows isn't perfect, and has rough edges. We had about two months to land a workable solution, so there are rough seams in here we had no time ot work over.

Rendering Is Bursty, and Bursty Is Hard

Render farms have a shape that's familiar to anyone who's run capacity for a spiky workload: baseline demand most of the year, and punishing peaks before a deadline. And here is a rough walk-through of the proposed shape of the solution:

• Artists submit render jobs on-prem.

• Scheduler decomposes each job into independent per-frame tasks.

• During peak load, compute bursts into AWS.

• Most cloud workers run on Spot instances to control cost.

• Assets live in S3.

• FSx for Lustre sits in front of s3 as a shared, high-throughput fs so hundreds of workers aren't all hammering S3 for the same textures at once.

• Finished frames go back to S3, and a sync service pulls them home to the studio.

Why Rendering Is One of the Best Workloads for Spot

Spot instances are cheap because AWS can reclaim them with few minutes' notice. That might be a dealbreaker for a lot of workloads - but rendering is close to the ideal case for it, and for three reasons:

1. Embarrassingly parallel. Each frame renders independently.

2. Stateless. A worker holds no state that matters beyond the frame it's currently on.

3. Idempotent. If a frame gets interrupted mid-render, the worst case is that it gets re-rendered.

Workers poll the instance metadata endpoint (169.254.169.254) at a fixed cadence for interruption notices. On a warning, a worker attempts to

• stop requesting new frames

• uploads whatever it has to the persistent storage, and

• finishes the one it's on if time allows, and

• terminates cleanly if possible

A small baseline of On-Demand ec2 instances keeps the farm going.

The fleet itself was built with CloudFormation and Auto Scaling Groups, using Packer-built golden AMIs - bake as much as possible into the image, let cloud-init handle registration, and workers are render-ready within minutes. Fast, and easy to reason about - which is exactly what you want for something that scales up and down constantly.

The Scheduler: Where the First Iteration Fell Apart

The scheduler is the control plane: job submission, batch decomposition, frame assignment, retries, completion tracking.

Workers are the data plane: they render and push to S3. Separating these cleanly matters, because it means the scheduler can hiccup for a few seconds without a single worker noticing.

But the first version of this design had a problem that's easy to miss until someone points it out: making the scheduler highly available doesn't help if all the state lives in the scheduler's memory. All you've done is move the single point of failure somewhere less visible.

The fix was to push state into the database and treat the scheduler as disposable:

• PostgreSQL on RDS Multi-AZ became the source of truth.

• Jobs, Frames, and Workers as first-class tables.

• Frames move through an explicit lifecycle: PENDING → RUNNING → COMPLETED / FAILED → PERMANENTLY_FAILED.

Once frame state lives in the database instead of in a process's memory, the scheduler itself becomes stateless and replaceable - which is what actually makes HA possible.

The Queue Bottleneck Nobody Notices at Small Scale

The way our scheduler was handing out frames:

SELECT * FROM frames WHERE status = 'PENDING' LIMIT 1 FOR UPDATE;

This works fine with ten workers. With hundreds, it becomes a lock-contention nightmare - workers queue up behind each other just to claim a single row, and the scheduler's effective throughput craters exactly when you need it most.

The fix is FOR UPDATE SKIP LOCKED (available since Postgres 9.5), combined with batched claims - a worker asks for 5–10 frames at a time instead of one, which cuts the number of round-trips by an order of magnitude. Beyond a certain fleet size, even this stops being enough, and the honest answer is to move the queue to something built for it - Redis, SQS, or Kafka. Postgres-as-a-queue is a fine choice until it isn't - but we never got to re-implement this portion.

Leader Election Has a Split-Brain Problem Too

Multiple scheduler instances need to agree on who's actually in charge, which is a classic use case for a Postgres advisory lock. But leader election alone doesn't prevent split-brain: if a leader stalls (GC pause, network blip) and a new one is elected, the old leader can wake up and keep assigning frames, unaware it's been replaced.

The fix is epoch fencing. Every elected leader gets a monotonically increasing epoch number, and every frame assignment carries that epoch. A stale leader with an old epoch simply can't overwrite assignments made by a newer one. It's a small mechanism, but it's the difference between "we have leader election" and "we have correct leader election."

The Control Loops We Missed

Initially we only (and embarrassingly) only designed the "happy-path" - the case where everything succeeds, and learnt it the hard way that a real system needs to answer: what happens when it doesn't?

The Frame Reaper. A worker can die mid-frame - Spot reclamation, a kernel panic, a network partition - and if nothing is watching, that frame sits in RUNNING forever, invisibly stuck. A reaper loop runs every 30 seconds, checks worker heartbeats, and flips stale RUNNING frames back to PENDING with an incremented retry count.

Poison frame handling. Some frames are just bad - a corrupt texture, a scene file that crashes the renderer. Without a cap, a poison frame retries forever and quietly eats capacity. max_retries = 5, then PERMANENTLY_FAILED and a notification to the artist who submitted it.

Priority that's actually enforced. Priority existed early on as a column with no teeth - nothing in the scheduler actually respected it. Fixing that meant an explicit ORDER BY priority in the claim query, plus a reserved fraction of workers for background jobs so low-priority work doesn't starve indefinitely.

None of these are exciting to design. All three are the difference between a system that survives a bad week and one that needs a human babysitting it.

Storage: Know What Your Source of Truth Actually Is

S3 is durable storage. FSx for Lustre is not - it's a performance cache sitting in front of S3, and conflating the two is a fast way to lose data. Workers writing to /mnt/lustre/fs/job/seq/scn/assets/<hash>/ does not mean that data exists in S3 unless it's explicitly exported. The production-safe pattern: workers write outputs directly to S3, and Lustre is used almost exclusively for reading input assets, avoiding the failure mode where 200 workers each try to pull the same 30 GB asset set independently.

Prewarming followed the same "define it operationally or it's not real" pattern. "Prewarm the cache" isn't a plan - it became: the scheduler builds an asset manifest for a job, a dedicated prewarm worker walks that manifest, and simply reading each file forces Lustre to pull it from S3 ahead of time. Job state gained an explicit PREWARMING stage so the system's state machine matches what's actually happening physically.

Worker Identity: Simpler Beat Fancier

The first pass at worker authentication was mTLS - which sounds appropriately serious for a distributed fleet, until you have to answer "how do we issue, rotate, and revoke certificates for autoscaling workers that live for twenty minutes?" That's a real operational burden for a workload that doesn't need it.

The simpler and more realistic answer: workers authenticate using their EC2 Instance Identity Document, which the scheduler validates against AWS's signature, with TLS handling transport security. It's a good example of a broader principle - the "more secure-looking" option isn't automatically the right one if it introduces lifecycle management you don't actually need.

Hybrid Networking and the Cost of Forgetting Bandwidth

The studio connects to AWS over Direct Connect with VPN as a fallback, and results flow AWS → S3 → a studio-side sync service (workers never write directly to studio storage). The detail that's easy to miss: a large enough result-sync job can saturate Direct Connect and start starving control-plane traffic - the very traffic your scheduler and workers need to keep functioning. The fix is unglamorous but necessary: bandwidth limits and bulk sync windows, with control-plane traffic always prioritized.

Autoscaling, Boot Storms, and Why 99.9% Doesn't Compose

At fleet scale, synchronized behavior becomes its own failure mode - hundreds of workers booting at once and hammering the same registration endpoint or polling loop at the same moment. Random startup jitter and frame batching smooth this out. Scaling itself needs hysteresis and cooldowns, or you end up oscillating capacity up and down in response to noise.

The broader lesson, and probably the most transferable one in the whole exercise, is about availability math. Five services each running at 99.9% don't combine to 99.9% - they compose multiplicatively:

0.999^5 ≈ 99.5%

Every synchronous dependency you add is a tax on your overall availability, and microservice architectures pay this tax constantly through added network hops, retry storms, and fan-out amplification. Circuit breakers exist specifically to stop retry storms from compounding a small failure into a large one - instead of hammering a struggling downstream with retries, the circuit opens and requests fail fast, protecting the thing that's already in trouble.

It's also, frankly, a solid argument for modular monoliths as a starting point: clear module boundaries, no network calls between them, simpler deployments, and - critically - higher baseline availability, because you haven't yet introduced the dependencies that erode it. Split into microservices when there's a real reason to, not by default.

What This Exercise Was Actually Teaching

Strip away the render-farm specifics and a handful of principles remain, and I think they generalize to basically any distributed system:

1. Separate control plane from data plane. If task state is durable, the scheduler can fail for a few seconds and nothing downstream notices.

2. Know your source of truth, explicitly. S3 was durable; FSx was a cache. Confusing a performance layer for a durability layer is how data quietly disappears.

3. Every control loop matters. The happy path is not a design - it's a slide. Reapers, retry limits, poison-record handling, and bandwidth governors are what make a system survive contact with reality.

4. Distributed systems are a series of tradeoffs, not a series of best practices. Performance vs. resilience, cost vs. availability, simplicity vs. scale - the right answer changes with fleet size, and a good design says when it changes, not just what the current answer is.

5. Availability is architectural, not aspirational. You don't get high availability by choosing highly-available components. You get it by reducing synchronous dependencies, placing redundancy where it actually matters, and designing explicitly for degradation instead of hoping it never happens.

What started as "how do I phrase a resume bullet about a hybrid render farm" turned into something closer to a staff-level design review - not because the technology got more exotic, but because the questions got sharper. Every fix above came from someone asking "okay, but what happens when this fails?" one level deeper than the previous draft had gone. That's really the whole job.

While managing our internal K8s platform I've come across multiple instances where the onboarding team(s) haven't really figured out the workload resource allocation math correctly thereby leading to unexpected results.

For instance when we deploy an app. to K8s, we might think we have the resource calculations perfectly figured out. If we have a 2Gi memory quota in our ns, and the deployment needs 1.8Gi of memory, everything should work smoothly, right? Unfortunately and possibly no as K8s has a few hidden math traps that frequently break deployments and cause pods to get stuck. Here is exactly how K8s handles memory limits and scheduling behind the scenes.

The Rolling Update Issue

The very first mistake people make is looking at a deployment's steady state. If you multiply your pod's memory limits by the number of replicas, you get your normal footprint. However, K8s rarely runs exactly that number of replicas during an update. By default, deployments use a rolling update strategy with a setting called maxSurge. This means it would temporarily spins up extra new pods before killing the old ones to prevent downtime. If your resource quota is tuned exactly to your normal running capacity, the rolling update will instantly hit the limit wall, and your new pods will be blocked. YOu might have to stream the ns specific events to figure this one out -
kubectl get events --sort-by=.lastTimestamp

Init Containers Twist the Math

Things could also get more interesting when you add init containers into the mix. Because init containers run Serially and finish before the main app. starts, they never run at the same time as your app. Because of this, K8s calculates your pod’s memory requirements using an "effective request" formula. The scheduler compares the sum of all your regular app containers against the single highest resource request among your init containers. Whichever number is bigger becomes the official memory requirement that the scheduler uses to find a node.

The Init Container Excessive Reservation Problem

This scheduling logic creates a strange side effect that might be called a "ghost reservation." If you have a heavy init container that needs 1Gi of memory to run a quick database migration, and an app container that only needs 100Mi, the scheduler permanently logs that pod as needing 1Gi of memory. Even after the init container finishes and hence is not using that memory, the scheduler does not give that space back to the node. That 1Gi remains blocked off for the entire lifecycle of the pod, which can easily cause your cluster nodes to look fuller when they are actually sitting idle. Take a look at this discussion for more information.

Memory Backed volumes count towards Memory Limit

The memory limit for the Pod or container can also apply to pages in memory backed volumes, such as an emptyDir. The kubelet tracks tmpfs emptyDir volumes as container memory use, rather than as local ephemeral storage.

Only came across this once one of our clients suffered a major outage unintentionally due to the fact (more to follow later in a separate blog post).

As medium: Memory leverages tmpfs (RAM) under the hood, writing to that directory allocates pages in the the page cache whose usage is is tracked by the kubelet under the container memory cgroup, it might create a series of catastrophic blind spot.

Limits vs Requests for the Scheduler

This often has been a source of constant confusion on our platform. yet another piece of puzzle is understanding what the scheduler actually cares about. The kube-scheduler completely ignores your memory limits when deciding where to place a pod; it only looks at memory requests.

The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node.

However, if you specify a memory limit for your container but forget to define a request, k8s automatically sets the request equla to your limit. In that specific scenario, your limit indirectly controls your scheduling, which can accidentally trigger more reservation than actually being used leading to wasted resources due to reduced allocatable capacity, or push your deployment past your ns hard resource quota silently.

Silent Danger of CPU Throttling

Another issue we've observed sneaking up silently on dev. teams is this one. While breaking a memory quota results in an immediate, loud OOMKilled error, misconfiguring your CPU limits creates a much more frustrating, silent failure.

CPU is a compressible resource.

When your app. hits its CPU limit, K8s doesn't kill the pod; it throttles it by slowing down its processing time. This causes your application to become incredibly slow and unresponsive. Because the app stops responding in time, its own livenessProbe or readinessProbe health checks will begin to fail. K8s will then assume the container is dead and trigger a cascading loop of unneeded restarts, all because a strict CPU quota starved the application of the processing power it needed to answer its own health checks.

K8s networking conversations these days seem to be dominated by discussions around Service meshes, and eBPF, yet many clusters today still rely on kube-proxy, and underneath kube-proxy lies a less glamorous but equally important question:

Should your cluster be using iptables or nftables?

If you're a platform engineer running Kubernetes at any reasonable scale, this isn't merely an implementation detail. It directly affects how efficiently your cluster handles Svc routing, endpoint updates, rolling deploys, and autoscaling events.

What follows is my understanding of the concepts learnt the hard way while dealing with running on-prem k8s platforms at scale -

A Simple Kubernetes Service

Imagine a Service with three backend Pods:

Svc A (10.96.0.10:80)

    ├── Pod 1 (10.244.1.10)
    ├── Pod 2 (10.244.2.10)
    └── Pod 3 (10.244.3.10)

When a client sends traffic to the Service VIP:

client
   |
   v
10.96.0.10:80

the kernel must answer a simple question:

Which backend pod should receive this connection?

The answer depends on how kube-proxy programs the node.

How iptables Solves the Problem

Historically, kube-proxy implemented svc load balancing using iptables.

For every svc, kube-proxy creates a chain:

KUBE-SERVICES
    |
    └── KUBE-SVC-ABC123

Inside that chain are endpoint selection rules:

Rule 1 -> Pod 1
rule 2 -> Pod 2
Rule 3 -> Pod 3

In reality, these rules use probabilistic matching to distribute connections across backends.

Conceptually:

Connection arrives
      |
      v
Check Rule 1
      |
      +--> Match -> Pod 1

Otherwise

Check Rule 2
      |
      +--> Match -> Pod 2

Otherwise

Pod 3

The important detail is that iptables represents backend selection as a collection of rules.

That works well when you have a handful of svc's and endpoints.

Where Things Start Getting Interesting

Imagine:

1,000 Services
50 Endpoints per Service

Now kube-proxy may be managing tens of thousands of iptables rules.

For a packet destined to a Service, the kernel must first locate the matching Service rule and then evaluate randomised endpoint selection rules within that svc chain.

The lookup is more like:

Find Service
      +
Find Endpoint

However, both of those operations become increasingly expensive as the number of Services and endpoints grows.

The Bigger Problem Isn't Packet Forwarding

One of the most common misconceptions is that iptables struggles because packet forwarding becomes unbearably slow.

In practice, packet forwarding is rarely the primary issue as in a K8s clusters things are constantly changing:

• pods are created and destroyed

• Deployments roll forward

• HPA scales workloads up and down

• Nodes join and leave the cluster

Every endpoint change forces kube-proxy to reprogram iptables.

As endpoints are added and removed, kube-proxy repeatedly regenerates and reapplies large portions of its rule set.

This leads to:

• Longer synchronization times

• Higher kube-proxy CPU usage

• Large iptables-restore ops

• Slower propagation of endpoint updates

In large clusters, these effects become very noticeable.

Enter nftables

nftables was designed as the successor to iptables.

The key architectural difference is subtle but important.

iptables treats networking state as rules.

nftables treats networking state as data.

Instead of representing backend selection as a series of rules, nftables can store information using efficient kernel data structures such as sets and maps.

Conceptually:

Service A
    |
    └── Backend Set
          ├── Pod 1
          ├── Pod 2
          └── Pod 3

Rather than traversing chains of rules, the kernel can perform direct lookups into optimized structures.

Think of it like the difference between:

Searching a list

and

Looking up an entry in a hash table

The exact implementation apparently is more nuanced, but the above mental model more or less holds

Why This Matters for K8s ?

The biggest benefit of nftables isn't just that packets suddenly move faster - It is operational efficiency.

Suppose a new Pod becomes available:

Pod 4 added

With iptables, kube-proxy may need to regenerate portions of the ruleset.

With nftables, the kernel can often update a set or map entry instead.

Conceptually:

Before:
[Pod1, Pod2, Pod3]

After:
[Pod1, Pod2, Pod3, Pod4]

The packet processing logic remains unchanged.

Only the data changes.

This makes endpoint updates significantly cheaper.

Scaling Characteristics

From my experience, the distinction becomes more important as cluster size grows. Consider two environments.

Small Cluster like DEV/QA

50 Nodes
200 Services

I personally have not found very any meaningful difference as imo iptables remains perfectly serviceable.

Large Platform like STG/PROD/PERF

1,000+ Nodes
10,000+ Services
Tens of thousands of Endpoints

Now the operational overhead of maintaining large iptables rule sets becomes much more apparent. This is where nftables begins to show its strengths:

• Faster synchronization

More efficient updatesnftables vs iptables: A Kubernetes Platform Engineer's Perspective

Kubernetes networking conversations these days seem to be dominated by discussions around Service Meshes, eBPF, and XDP. Yet many production clusters today still rely on kube-proxy, and underneath kube-proxy lies a less glamorous but equally important question:

Should your cluster be using iptables or nftables?

If you're a platform engineer running Kubernetes at any reasonable scale, this isn't merely an implementation detail. It directly affects how efficiently your cluster handles Service routing, endpoint updates, rolling deployments, and autoscaling events.

What follows is my understanding of these concepts, learned the hard way while operating large on-prem Kubernetes platforms.

A Simple Kubernetes Service

Imagine a Service with three backend Pods:

Service A (10.96.0.10:80)

    ├── Pod 1 (10.244.1.10)
    ├── Pod 2 (10.244.2.10)
    └── Pod 3 (10.244.3.10)

When a client sends traffic to the Service VIP:

Client
   |
   v
10.96.0.10:80

the kernel must answer a simple question:

Which backend Pod should receive this connection?

The answer depends on how kube-proxy programs the node.

How iptables Solves the Problem

Historically, kube-proxy implemented Service load balancing using iptables.

For every Service, kube-proxy creates a dedicated chain:

KUBE-SERVICES
    |
    └── KUBE-SVC-ABC123

Inside that chain are endpoint selection rules:

Rule 1 -> Pod 1
Rule 2 -> Pod 2
Rule 3 -> Pod 3

In reality, these rules use probabilistic matching to distribute new connections across backends.

Conceptually:

Connection arrives
      |
      v
Check Rule 1
      |
      +--> Match -> Pod 1

Otherwise

Check Rule 2
      |
      +--> Match -> Pod 2

Otherwise

Pod 3

The important detail is that iptables represents backend selection as a collection of rules.

That works perfectly fine when you have a handful of Services and endpoints.

Where Things Start Getting Interesting

Now imagine a larger environment:

1,000 Services
50 Endpoints per Service

At this point, kube-proxy may be managing tens of thousands of iptables rules.

For a packet destined to a Service, the kernel must first locate the matching Service rule and then evaluate endpoint selection rules within that Service chain.

The lookup is conceptually:

Find Service
      +
Find Endpoint

Contrary to what some engineers assume, the kernel is not traversing every endpoint rule in the cluster. However, both Service lookups and endpoint selection become increasingly expensive as the number of Services and endpoints grows.

The Bigger Problem Isn't Packet Forwarding

One of the most common misconceptions is that iptables struggles because packet forwarding becomes unbearably slow.

In practice, packet forwarding is rarely the primary issue.

The real challenge is change.

A Kubernetes cluster is a constantly moving target:

• Pods are created and destroyed

• Deployments roll forward

• HPAs scale workloads up and down

• Nodes join and leave the cluster

Every endpoint change forces kube-proxy to reprogram iptables.

As endpoints are added and removed, kube-proxy repeatedly regenerates and reapplies large portions of its rule set.

This leads to:

• Longer synchronization times

• Higher kube-proxy CPU utilization

• Large iptables-restore operations

• Slower propagation of endpoint updates

In sufficiently large clusters, these effects become very noticeable.

Enter nftables

nftables was designed as the successor to iptables.

The key architectural difference is subtle but important.

iptables treats networking state as rules.

nftables treats networking state as data.

Instead of representing backend selection as a series of rules, nftables can store information using efficient kernel data structures such as sets and maps.

Conceptually:

Service A
    |
    └── Backend Set
          ├── Pod 1
          ├── Pod 2
          └── Pod 3

Rather than traversing chains of rules, the kernel can perform direct lookups into optimized structures.

A useful mental model is:

iptables  -> Search a list
nftables  -> Lookup in a hash table

The actual implementation is more nuanced than that, but the comparison is directionally accurate.

Why This Matters for Kubernetes

The biggest benefit of nftables isn't that packets suddenly move faster.

The bigger win is operational efficiency.

Suppose a new Pod becomes available:

Pod 4 added

With iptables, kube-proxy may need to regenerate portions of the ruleset.

With nftables, the kernel can often update a set or map entry instead.

Conceptually:

Before:
[Pod1, Pod2, Pod3]

After:
[Pod1, Pod2, Pod3, Pod4]

The packet processing logic remains unchanged.

Only the underlying data changes.

This makes endpoint updates significantly cheaper and reduces the amount of work required during periods of high endpoint churn.

Scaling Characteristics

From my experience, the distinction becomes more important as cluster size grows.

Small Clusters (Dev / QA)

50 Nodes
200 Services

I personally haven't observed a meaningful difference in day-to-day operations.

In environments of this size, iptables remains perfectly serviceable.

Large Platforms (Stage / Perf / Production)

1,000+ Nodes
10,000+ Services
Tens of Thousands of Endpoints

Now the operational overhead of maintaining large iptables rule sets becomes much more apparent.

This is where nftables begins to show its strengths:

• Faster synchronization

• More efficient updates

• Better scaling characteristics

• Reduced rule management complexity

The Real Takeaway

As platform engineers, it's mostly easy to get distracted by datapath benchmarks and packet-per-second numbers.

The more important question is:

how efficiently can my cluster adapt to change?

Modern Kubernetes environments are dynamic systems. Pods come and go constantly. Deployments roll all the time. Autoscalers react to traffic spikes, so basically - Endpoint churn is normal.

Viewed through that lens, nftables offers a cleaner and more scalable foundation than traditional iptables.

iptables treats Service routing as a collection of rules (if-elif-else construct) where as nftables treats svc routing as data (hash lookups)

That shift may sound subtle, but at k8s scale, it can make a significant difference.

And while kernel based xDP and eBPF-based solutions are increasingly becoming the industry's long-term direction, understanding the evolution from iptables to nftables remains essential for anyone building and operating modern Kubernetes platforms.

When you operate an on-prem K8s platform with a static node count that hasn't changed for months, a single dead node could have a ripple effect on the entire cluster, and the workloads being run causing a massive headache. Unlike the cloud managed K8s clusters, at tmies we don't have the luxury of auto-scaling groups silently spinning up fresh instances in the background. If a node drops, we lose hard capacity until someone physically intervenes ,and in case the clsuter is already under duress it only amplifies.

This is exactly what was experienced on the primary Perf cluster very recently. One of the dev teams deployed a distributed workload for QA with several enhancements including using memory backed emptyDir vols. as a ephemeral cache, which resulted in a cascading failure where the cluster performance was significantly degraded. There was no NotReady transition or graceful pod eviction - instantaneous lockups. Couldn't even SSH into some of the boxes.

Here is how a single volume mount config exploited a K8s accounting loophole (Issue #119611) to starve our cluster, and the logical steps we took to debug the app.

The Investigation: Step-by-Step

Step 1: The Hard Reset and Kernel Autopsy Since sshd was completely unresponsive, we had to hit the nodes via out-of-band mgmt to issue a hard power cycle. Once the node rejoined the cluster, my first instinct was to query Loki (via Grafana) for the kubelet logs just prior to the crash. We fully expected to see the standard K8s resource starvation chain-of-events -

• kubelet crossing its evictionHard threshold

• tainting node with MemoryPressure, and

• gracefully terminating BestEffort or Burstable pods first

Instead, the Loki streams had completely dropped the last few minutes before the crash—a classic indicator that the logging daemon itself was starved of CPU/memory. Pulling dmesg and journalctl -k directly from the local nodes revealed system-level chaos - kernel's OOMkiller had been invoked, but it wasn't just reaping misbehaving containers; it was indiscriminately killing core OS processes. The node didn't gracefully handle the memory pressure; the kubelet was completely ambushed before it could even calculate the usage and trigger an eviction loop.

Step 2: Isolating the Suspect Workload As the load-testing event was already previously advertised before hand we knew where the smoking gun might be pointing at. The workload deployment manifest included:

volumes:
  - name: cache-vol
    emptyDir:
      medium: Memory

Step 3: The Shared Quota Trap An emptyDir with medium: Memory tells K8s to mount a tmpfs (RAM disk) inside the container. Here is the critical detail that caught me off guard: tmpfs usage counts directly toward your container's mem limit.

• The memory limit for the Pod or container can also apply to pages in memory backed volumes, such as an emptyDir. The kubelet tracks tmpfs emptyDir volumes as container memory use, rather than as local ephemeral storage. When using memory backed emptyDir, be sure to check the notes below

It acts as a shared quota. FOr example, If you assign a container a mem limit of 1Gi, that must accommodate both the app's RSS/heap and any files written to the tmpfs volume. If the sum of the app mem and the tmpfs data exceeds the defined limit, the container might get OOMKilled.

Step 4: Discovering the Cgroup Loophole (The Root Cause) The devs hadn't set a sizeLimit on the volume. As the app was aggressively writing cache files, the shared quota was quickly breached, and the app was OOMKilled. Standard K8s behavior, right?

But here is where the K8s accounting loophole triggered the outage - when a container is OOMKilled and restarts, the pod remains on the node, and the data previously written to the tmpfs persists. However, when the new container spins up, the Linux kernel (possibly) often fails to account for that pre-existing tmpfs data in the new container's cgroup.

This data becomes "ghost mem." It consumes RAM but is invisible to the new cgroup limit. The app restarts, writes more files to the tmpfs, consumes its full mem limit again, crashes, and restarts. This loop continued until the entire node ran out of RAM and locked up. Now how did the kubelet not step in earlier to stop this madness is beyond me.

Learnings

K8s is incredibly powerful, but sometimes it expects you to provide strict guardrails. When you don't, features like mem-backed volumes become loaded weapons.

• It is advisable to enforce size limits for emptyDir volumes in new workloads (pods) using a policy mechanism such as ValidationAdmissionPolicy.

• Set a sizeLimit on mem-backed volumes. If you use medium: Memory, cap it. If a volume hits its sizeLimit, the kubelet gracefully evicts the pod instead of triggering an abrupt OOMKill.

emptyDir:
  medium: Memory
  sizeLimit: 500Mi

• Enforce hard mem limits. Set resources.limits.memory on the pod to ensure the kubelet and container runtime have a hard ceiling to enforce.

• Use Resourcequotas. At the platform level, enforce a ResourceQuota in every ns that mandates limits. This prevents teams from deploying unbounded apps in the first place.

• DON'T Rely on LimitRange for emptyDir sizing. While a LimitRange might apply default sizes to containers, it does not effectively prevent the mem overcommit scenario caused by unbound tmpfs volumes.

• DON'T Treat tmpfs like disk storage. Educate the devs that medium: Memory is RAM. It shares the same budget as the app's heap size. If an app needs a massive scratchpad, write to standard disk-backed storage instead.

Seeing is believing, so I wanted to build a demo to compare the GIL vs No-GIL impact on a concurrent setup. As i've been working on K8s controllers a bit those days, I made a multi-threaded benchmark that mimics a CPU-bound k8s Controller.

Why does the GIL exist anyway?

If you ever wrote multi-threaded Python, you have hit the Global Interpreter Lock (GIL).

It exists for one main reason: memory safety. CPython uses reference counting to manage memory. Every var., list, or dict tracks how many parts of your code point to it. When that count hits zero, Python end up frees the memory.

But if multiple threads try to update this counter at the same time, you get a race condition. This can crash your app or leak memory. To stop this, the creators of CPython added the GIL: a single giant lock. Only the thread holding the lock can run Python code.

This kept Python simple, but it created a big limit: even on a 32-core workstation, your multi-threaded Python runs on one core at a time.

PEP 703: The No-GIL Fix

To remove the GIL safely, PEP 703 made 3 big upgrades:

1. Biased Ref Counting: Objects are tracked differently depending on which thread made them. This stops CPU cache problems.

2. Thread-Safe Allocator: It uses mimalloc, a very fast concurrent memory allocator.

3. Fine-Grained Locks: It locks individual dicts and collections instead of locking the whole app.

The k8s Use Case

In the cloud-native world, many custom operators and controllers are written in Python (using tools like Kopf). They use threads to handle events from the k8s API server.

If your operator gets too many events at once, it starts threads to handle them. If those threads do CPU work—like parsing JSON, changing config keys, or making crypto hashes—they fight for the GIL. On standard Python, this lock fight can make your multi-threaded code slower than a simple sequential loop!

To prove this, I wrote a script to simulate this exact workload.

The Benchmark: k8s_bench.py

This script starts 4 threads to run 80k mock k8s tasks.

In each task, the worker must:

1. Decode a raw JSON string of a Pod manifest.

2. Mutate a key (injecting a unique uid).

3. Serialize it back to bytes.

4. Calculate a SHA-256 crypto hash to simulate config-drift checks.

import sys
import time
import threading
import json
import hashlib

def get_gil_status():
    try:
        enabled = sys._is_gil_enabled()
        return "ENABLED" if enabled else "DISABLED (No-GIL!)"
    except AttributeError:
        return "ENABLED (Legacy Interpreter)"

def simulate_k8s_reconciliation(worker_id, num_items):
    mock_pod = {
        "apiVersion": "v1",
        "kind": "Pod",
        "metadata": {"name": "nginx-ingress", "namespace": "ingress-nginx"},
        "spec": {
            "containers": [{"name": "nginx", "image": "nginx:1.25", "resources": {"requests": {"cpu": "100m"}}}]
        }
    }
    raw_json = json.dumps(mock_pod)
    
    hash_accumulator = 0
    for _ in range(num_items):
        parsed = json.loads(raw_json)
        parsed["metadata"]["uid"] = f"worker-{worker_id}-{_}"
        serialized = json.dumps(parsed).encode('utf-8')
        
        # CPU work: calculate SHA-256 hash
        hasher = hashlib.sha256()
        hasher.update(serialized)
        hash_accumulator += int(hasher.hexdigest()[:8], 16) % 1000
        
    return hash_accumulator

def run_benchmark():
    TOTAL_PODS = 80000
    NUM_WORKERS = 4
    PODS_PER_WORKER = TOTAL_PODS // NUM_WORKERS
    
    print("================================================================")
    print(f" PYTHON RUNTIME: {sys.version.split()[0]}")
    print(f" GIL STATUS:     {get_gil_status()}")
    print(f" WORKLOAD:       Reconciling {TOTAL_PODS:,} Pods via {NUM_WORKERS} tasks")
    print("================================================================\n")

    # Sequential Run
    print("Running Sequential (1 Thread)...")
    start_time = time.time()
    start_cpu = time.process_time()
    
    seq_acc = 0
    for i in range(NUM_WORKERS):
        seq_acc += simulate_k8s_reconciliation(worker_id=i, num_items=PODS_PER_WORKER)
        
    seq_real_time = time.time() - start_time
    seq_cpu_time = time.process_time() - start_cpu
    
    print(f"--> [SEQ] Real Time: {seq_real_time:.4f}s | CPU Time: {seq_cpu_time:.4f}s (Sum: {seq_acc})\n")

    # Multi-Threaded Run
    print(f"Running Multi-Threaded ({NUM_WORKERS} Threads)...")
    threads = []
    
    start_time = time.time()
    start_cpu = time.process_time()
    
    for i in range(NUM_WORKERS):
        t = threading.Thread(target=simulate_k8s_reconciliation, args=(i, PODS_PER_WORKER))
        threads.append(t)
        t.start()
        
    for t in threads:
        t.join()
        
    multi_real_time = time.time() - start_time
    multi_cpu_time = time.process_time() - start_cpu
    
    print(f"--> [MUT] Real Time: {multi_real_time:.4f}s | CPU Time: {multi_cpu_time:.4f}s")
    print(f"--> [MUT] Speedup: {seq_real_time / multi_real_time:.2f}x\n")

if __name__ == "__main__":
    run_benchmark()

Analyzing the Results

When you run this on standard Python vs. No-GIL Python, the numbers show a huge difference.

Standard Python (GIL Enabled)

• Real Time: The sequential run takes about 10s. The multi-threaded run also takes around ~10s (sometimes slower due to lock overhead).

• OS CPU Time: Both runs show almost the same CPU time as Real Time. Only one core worked at any second. Threads were just taking turns.

• Speedup: 1x

Free-Threaded Python (GIL Disabled)

• Real Time: The sequential run takes 10s, but the multi-threaded run finishes in just under 3s - hooray it works!

• OS CPU Time: Here is the math proof—the CPU time spikes to almost 11s even though the clock only ticked for 3s. This is because the OS adds up CPU cycles of all 4 cores working at the exact same time.

• Speedup: ~=3.5 - 3.8x (almost perfect scaling on a 4-core CPU).

What about Network (I/O) tasks?

While testing, there was an additional thought:

What if our controller was mostly waiting on API n/w calls?

As it turns out, n/w-bound apps don't really need No-GIL.

Standard Python automatically releases the GIL whenever a thread is waiting on n/w I/O (like waiting for a response from etcd). While Thread A waits for the n/w, Thread B grabs the GIL and runs. Since the CPU is mostly waiting, standard Python handles thousands of concurrent threads just fine.

But once the n/w data arrives, you still have to parse the JSON. In large scale clusters where you parse megabytes of JSON every second, that parsing step is heavy CPU work. In that hybrid scenario, a No-GIL build helps your threads parse those payloads in parallel across all your CPU cores.

Wrap Up

Testing the free-threaded interpreter was a great way to refresh my concurrency skills. Python is no longer locked to a single track; it can finally use modern multi-core CPUs natively- but of-course with strings attached :-)

If you build high-throughput tools, k8s operators, or data pipelines in Python, the No-GIL build is a massive step forward.

When dealing with read-your-own-writes (RYOW) issues in a read-heavy app, the first suggestion is almost always to just turn on synchronous replication.

There is a (wide-spread?) misconception that If the primary db waits to ack the write until all replicas confirm they have saved the data, the RYOW problem gets solved. BUT, it is a massive trap, and exposes a fundamental misunderstanding of what "synchronous" actually means to the db engine.

In standard sync replication, the primary only waits for the replica to confirm it has written the transaction to its Write-Ahead Log (WAL) on disk.

It guarantees durability, not visibility.

The replica has safely saved the data, but it has not necessarily applied it to the actual tables in memory yet.

If a read query hits that replica 1ms after the ack returns, the user will still see stale data because the replica's local apply process hasn't caught up.

To guarantee true visibility, you have to force the primary to wait until the replica has fully applied the transaction to memory (like using remote_apply in Postgres).

Configuring this extreme level of strict sync repl. not just adds latency penalties to the WRITE OP., it also ties write availability directly to the slowest read replica. If a single replica goes down, or the node crashes, the primary db literally stops accepting writes.

A minor read staleness issue suddenly transforms into a complete system outage.

Solving this at the db replication layer is the wrong approach. The routing layer is where this needs to be handled. There are two primary architectural patterns to fix this without tanking write performance.

1. Sticky routing (simpler)

When a user makes a write, a timestamp is dropped in their session cache. For a short window (say, 5 seconds), the app routes all of that specific user's reads directly to the primary db. Everyone else keeps hitting the replicas.

# Write operation
def update_profile(user_id, user_data):
    primary_db.write(user_data)
    # Drop a flag in cache that expires in 5 seconds
    cache.set(f"ryow_flag_{user_id}", True, ttl=5) 

# Read operation
def get_profile(user_id):
    # Check if the user recently wrote data
    if cache.get(f"ryow_flag_{user_id}"):
        return primary_db.read(user_id) # Safe fallback
        
    return replica_db.read(user_id) # Standard path

Pros:

• Incredibly easy to implement.

• Most frameworks support it out of the box or with simple middleware,

• requiring zero changes to db engine internals.

Cons:

• It relies on arbitrary time windows, which is essentially just hoping for the best, and If repl. lag spikes past the defined window, the issue returns.

• It also puts unnecessary read load on the primary db for those 5 seconds, even if the replicas caught up in 10ms.

• gets worse with the number of WRITE ops scaling.

• caching SPOF

2. LSN tracking (complicated, but precise way)

Every write generates a log sequence number (LSN). The app stores that LSN in a cache, and queries it first in case of a read operation, essentially baking in the logic not to answer the query from a RR until it has processed the WAL up to this specific LSN. If the replica is too far behind, the app falls back to the primary db.

# Write operation
def update_profile(user_id, user_data):
    # Db returns the exact log position of the write
    lsn = primary_db.write_and_get_lsn(user_data) 
    cache.set(f"user_lsn_{user_id}", lsn)

# Read operation
def get_profile(user_id):
    required_lsn = cache.get(f"user_lsn_{user_id}")
    replica_lsn = replica_db.get_current_lsn()

    # Mathematically guarantee the replica has applied the WAL
    if replica_lsn >= required_lsn:
        return replica_db.read(user_id)
        
    return primary_db.read(user_id) # Fallback if replica is lagging

Pros:

• Mathematically perfect consistency.

• Stale data is never served, and replica usage is maximized because routing to the primary db only happens if absolutely necessary.

Cons:

• Massive engineering effort.

• It tightly couples app code to specific db engine internals,

• and almost no standard ORMs support this natively.

• the queries to the RR doubles.

• caching SPOF

NOTE: There might be other (efficient) approaches too in addition but these 2 are the ones that I personally have come across.

The takeaway

Sticky routing often wins simply because the engineering overhead of building custom LSN trackers is rarely worth it for majority of RYOW apps.

To me accepting a slight bump in primary db load to keep app code simple is usually the right trade-off while keeping the primary db highly available.

Next time the db layer scales out, remember the golden rule of infra: never let read capacity compromise write availability.

Back then I was doing a lot of perf-related work and benchmarking, and I remember being confused by something I'd seen countless times:

$ time ./my_program

real    2.1s
user    6.4s
sys     0.3s

How could a program that finished in just over 2 seconds spend 6 seconds on the CPU?

I dug into it at the time and made a few notes. Reading them again was a nice refresher, so I cleaned them up a bit and thought I'd share them here. Maybe they'll save someone else a trip down the same rabbit hole, or at the very least serve as a refresher to aa future me :-)

What does time actually measure?

The linux time command prints three values:

real
user
sys

I used to look at them without thinking much about what each one actually meant.

real

This one's straightforward as It's just the wall-clock time.

start the program -> RUN -> stop the program.

And whatever a stopwatch would show is real.

It includes everything:

• CPU execution

• waiting on disk

• waiting on the network

• sleeping

• scheduler delays

• lock contention

Basically, the total time the process existed.

user

This is where things started making more sense.

user is the amount of CPU time spent executing your application's code.

Think any runtrime like Go, Rust, Python, Java, C++, etc. But do notice I said CPU time, not elapsed time.

sys

sys is also CPU time, but it's time spent inside the Linux kernel on behalf of your process.

Typical examples are:

• read()

• write()

• send()

• recv()

• page faults

• filesystem work

So why can user be larger than real?

This was the part that finally made it click for me.

Imagine your machine has four CPU cores.

Your program spins up two worker threads and both stay busy for two seconds.

Core 0  ████████ 2 sec

Core 1  ████████ 2 sec

Core 2  idle

Core 3  idle

The program still finishes after two seconds.

So:

real = 2 sec

But Linux accounts CPU usage separately for each core.

Core 0 = 2 sec
Core 1 = 2 sec

Total CPU time = 4 sec

Which means this is perfectly valid:

real    2 sec
user    4 sec

CPU time is accumulated across every CPU executing your process.

What about everything else running on the machine?

Another question I had was whether time includes CPU used by other applications.

Suppose your workstation has 32 cores.

Chrome is open.

Docker is busy.

Maybe a few k8s clusters are running locally.

None of that matters.

time only reports CPU consumed by your process (and usually any child processes that it creates and waits for).

Everything else is ignored.

Python's process_time()

Python has something very similar.

import time
start = time.process_time()
do_work()
end = time.process_time()

This measures CPU time for the current process.

For example:

time.sleep(5)

will roughly look like this:

Wall time ≈ 5 sec
CPU time ≈ 0 sec

because the process is sleeping, not executing instructions.The same idea applies to network I/O. f an HTTP request takes 500 ms but your process only spends 4 ms actually executing code, then:

Wall time = 500 ms
CPU time = 4 ms

Which timer should I use?

Use the timer that matches the question you're asking. For example, if you're trying to benchmark how much CPU your code consumes:

time.process_time()

makes sense.

If you're measuring request latency, API response times, or anything the user experiences:

time.perf_counter()

is usually the better choice as they're measuring two different things.

Final thoughts

Looking back, this wasn't a particularly difficult concept. I just never had a reason to stop and think about what Linux was actually reporting.

It's funny how some tools become muscle memory. we use them almost every day, but don't always understand what they're doing underneath.

Anyway, I figured these old notes were worth keeping around. If you've ever looked at output like this:

real    2.0
user    7.3
sys     0.2

and wondered if something was broken, it probably isn't.

It usually just means your program managed to keep multiple CPU cores busy at the same time.