ETCD is the single source of truth for a Kubernetes cluster. Every Node, Pod, Secret, ConfigMap, CRD, and RBAC rule lives there. Lose etcd on all control plane nodes and you have lost the cluster - the entire desired state is gone. So on a K8s platform two things have to be true -
That second half is where most write-ups I've come across quietly wave their hands. They show you kubectl cp from a backup Pod, which is lovely right up until the day you actually need it, when the API server (API) is down and kubectl returns nothing but connection errors.
This post is Part 1, how we take snapshots and how we get one back with the cluster fully dead. Part 2 covers the restore itself, which is where the real landmines are.
The backup
A k8s CronJob in a dedicated namespace runs every 1h and calls etcdctl snapshot save against the live cluster. This gives us a recovery point objective (RPO) of roughly 1h. That is a deliberate tradeoff based on our internal calculations; a tighter RPO means more frequent snapshots and more storage churn, and you should size it to how much state your cluster can afford to lose.
The snapshot lands on a NetApp TRIDENT NFS-backed PVC with a sidecar pruning anything older than 30 days.
The PVC itself is normally reached only through a small validation Deployment that we keep scaled to 0 from within the cluster
Or NFS mountable on one of the master nodes for a more direct and traditional access.
Two facts about that .db file drives everything downstream:
It is a full, point-in-time (PIT) copy of the entire keyspace. Restoring it rewinds the whole cluster to that instant.
It contains every Secret in plaintext. etcd at rest is usually unencrypted, so the snapshot is exactly as sensitive as the cluster it came from. Treat every copy as mode 0600 and delete it the moment you are done.
Retrieval: part that actually matters
Getting the snapshot back has two very different modes, and the gap between them is the whole point of this post.
Mode A, the cluster is still up. This is the HAPPY path.
Mode B, the cluster is down. The SAD path, because Mode A depends on a working API and kubectl, which is precisely what a real DR event takes away. The good news though is that the backup lives on an NFS export, and we skip k8s entirely and mount the export directly on a surviving master, which still has L3 reach to the storage data LIF.
Mode C, offsite object storage. We also ship every snapshot to blob storage, which is the cleanest retrieval when it works, one authenticated pull with a CLI
<cloud> storage blob download --container etcd-backups \
--name etcd-snapshot-<...>.db --file /root/etcd-snapshot.db
The catch is credentials**. In our outage the blob access keys themselves were "lost" due to the extent of the outage, which is exactly why we fell back to the NFS mount in Mode B.**
The blob failure taught a real lesson - one retrieval path at times may not be enough. So we promoted the NFS mount from a manual break-glass step to a first-class, automated fetch mode that sits alongside the simple blob download - two independent paths.
Mounting the backup export on a master
You need two coordinates, the NFS server address and the export path. Record them now, while the cluster is healthy, because you cannot ask the PV for them once the API is gone:
PV=$(kubectl -n ckp-cluster-jobs get pvc ckp-cluster-jobs-etcd-backup-data -o jsonpath='{.spec.volumeName}')
kubectl get pv "$PV" -o jsonpath='{.spec.nfs.server}{" "}{.spec.nfs.path}{"\n"}'
Stash those in the Ansible per-cluster inventory (group_vars) so they are already on disk before you need them. During DR, the manual form is just a read-only mount, copy the newest file off, unmount:
mkdir -p /mnt/etcd-backup
mount -t nfs -o ro,nfsvers=4.1 <server>:<export> /mnt/etcd-backup
ls -ltr /mnt/etcd-backup
cp /mnt/etcd-backup/etcd-snapshot-<...>.db /root/etcd-snapshot.db
umount /mnt/etcd-backup
Read-only is a safety guardrail - the mount physically cannot modify or delete your backups, which is exactly the guarantee you want when you are already having the worst day of your quarter.
Verify before you trust
A snapshot you have not checked is a guess. Before you build a restore around it, confirm it is intact and confirm it is the point in time you think it is:
etcdutl snapshot status /root/etcd-snapshot.db -w table
That prints the hash, total keys, and the revision. The revision matters, it tells you roughly how recent the snapshot is, and it is what a PIT restore lands on.
If this command errors, the file is truncated or corrupt, and you want to know that now, not halfway through a restore.
Automation: the etcd-snapshot-fetch role
Doing the mount dance by hand at 3am invites mistakes, so we wrapped it in an Ansible role, etcd-snapshot-fetch, driven by a playbook that targets one master (master1 by default):
ansible-playbook -i inventory/<cluster> etcd-snapshot-fetch.yaml \
-u <user> -kK -b -e target_cluster=<cluster>
The flow is:
Assert the target host actually belongs to the named cluster becasue the last thing we'd want in this scenario is to have 2 outages instead of one :-)
Mount the export read-only on the master.
Find the newest etcd-snapshot-*.db, or a specific one you name for a PIT restore.
Copy it off the read-only mount to local disk on the master, then fetch it to the Ansible control host.
Always unmount and clean up, in an always: block, so the export is never left mounted even if a step fails.
The output is a verified snapshot on your control host at 0600, ready to hand to the restore. Notice the security posture end to end.
What Part 1 buys you
You now have a verified snapshot in hand, retrieved through a path that does not depend on the cluster being alive. That independence is the entire point. A backup you can only reach through the thing that just died is not a backup, it is a hope.
There is one more habit worth building - rehearse it. A backup nobody has ever restored is Schroedinger's backup, simultaneously fine and useless until you open the box, which is a good segue into Part 2 as that is where we open the box, and where restoring onto a 3-master cluster goes wrong in ways that surprised us.
Sources
