# Surviving A Complete ETCD Outage, Part 2: Restoration

In [Part 1](https://abhishek-pareek.hashnode.dev/surviving-a-complete-etcd-outage-part-1-snapshots-and-getting-them-back-when-the-cluster-is-gone) we got a verified [ETCD](https://etcd.io/) snapshot back onto an [Ansible](https://ansible.com) control host, even with the cluster fully down.

I've come across many a blog posts describing ETCD restoration process on a single node, but not on a full 3 master setup which is what everyone in production uses at the very least. This post is the failure catalog we wish we had read first.

## Firstly: Identify the 'Failure Mode'

A restore rewinds the whole cluster to the snapshot instant, discarding every write since. That is fine when you have lost everything, but it might well be an unnecessary data loss you've inflicted on yourself in other cases.

A 3-member **etcd cluster needs a majority, 2 of 3, to serve writes**. So triage before you touch anything as getting it right is half the value:

*   **1 master down.** Quorum retained (2/3), cluster fully operational - Remove the dead member, add a fresh one.
    
*   **2 masters down.** Quorum lost (1/3) - **read-only,** but the survivor still holds current data. Do NOT restore. Recover quorum on the survivor with `--force-new-cluster`, then re-add members.
    
*   **All 3 down.** No surviving data. **This, and only this, is a snapshot restore.**
    

## The trap

The obvious idea for all-3-down is, "restore the snapshot on each master". It is wrong, and it fails in a way that wastes an hour of your time eating into your precious RTO. `etcdutl snapshot restore` **mints a brand new cluster ID every time it runs** - Restore on all three and you get three separate one-member clusters, three different cluster IDs, **that will never form a quorum together.**

> The symptom is master1 endlessly probing the other two as peers, "connection refused" or a cluster ID mismatch in the logs, and no leader, ever.

## Correct shape: restore once, then add

The procedure that actually works, and the one our DR plan mandates, is:

1.  Restore the snapshot exactly once, on **master1**, as a single-member cluster.
    
2.  Add master2, then master3, as new members with `etcdctl member add`. They start with an empty data dir and replicate from master1.
    

One restore followed by two live joins. Concretely, once master1 is up as a single healthy member, each joiner is a register-then-start, run from master1:

```bash
# 1. register the joiner in the cluster (from master1)
etcdctl member add master2-<cluster> --peer-urls=https://10.0.0.2:2380 \
  --endpoints=https://10.0.0.1:2379 $CERTS
# 2. on master2: set --initial-cluster-state=existing and --initial-cluster to the
#    growing member list in its etcd manifest, empty /var/lib/etcd, then start etcd
# 3. confirm from master1 that master2 is 'started' before you touch master3
etcdctl member list -w table --endpoints=https://10.0.0.1:2379 $CERTS
```

We wrapped the whole sequence in an Ansible role, `etcd-restore`, so the real run is one command:

```bash
ansible-playbook -i inventory/<cluster> etcd-restore.yaml \
  -u <user> -kK -b -e target_cluster=<cluster> -e etcd_snapshot_db=/root/etcd-snapshot.db
```

However the manual steps still matter, because break-glass is real, and because every one of the following issues is something the automation had to learn to survive.

```bash
etcdutl snapshot restore /root/etcd-snapshot.db \
  --name master1-<cluster> \
  --initial-cluster master1-<cluster>=https://10.0.0.1:2380 \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd/.restore-tmp
mv /var/lib/etcd/.restore-tmp/member /var/lib/etcd/member
rmdir /var/lib/etcd/.restore-tmp
```

On etcd 3.6, `etcdctl snapshot restore` was removed, so the offline restore must use `etcdutl`.

`etcdctl` is still what you use for the online `member add` and `endpoint health` later. You need both binaries, matched to the cluster's etcd version.

## Landmine 1: the member name must match the manifest

The `--name` and the `--initial-cluster` key you restore with have to exactly equal the `--name` in master1's etcd static Pod manifest, which is the node's full hostname, not the literal "default" that most examples use. Restore with the wrong name and the member never lines up with its peer URL mapping, and the cluster will not form. Read the real value straight from the manifest, do not guess.

## Landmine 2: --initial-cluster-state=existing must actually be present

When you add master2 and master3, each joiner's manifest needs `--initial-cluster-state=existing` plus an `--initial-cluster` listing every member up to and including itself. Miss the `state=existing` line and etcd defaults to `new`, so the joiner bootstraps its own cluster instead of joining, and you are back to the cluster ID mismatch from the trap above.

## Landmine 3: you cannot add a member to an unhealthy cluster

etcd refuses `member add` if the current cluster is not healthy, with "etcdserver: unhealthy cluster". After you add master2, it has to replicate the full keyspace before it is a healthy voter, and that can take tens of seconds for a large DB. Add master3 during that window and the request is rejected, because going from 2 voters to 3 while only 1 is healthy would break quorum. So you gate each join on the current cluster being fully healthy first, and you add members strictly one at a time.

The same rule bites during any restart. Never bounce an etcd member unless the other two are healthy, or you drop below quorum. We learned this the hard way when an optional "tidy the manifests" step restarted a member while another was still down, and froze the cluster.

## Landmine 4: Do not boot stale data

If master1's old data dir survives, because a wipe was skipped or incomplete, the move-into-place step can be silently skipped and etcd boots the previous cluster's multi-member data, then hangs probing peers that no longer exist. Guarantee the data dir is empty before you restore, do not rely on a "create only if missing" guard to protect you.

## The revision-bump problem

There is a subtle failure that survives everything above, and only the official etcd docs really mention it. It is about [revisions](https://github.com/etcd-io/etcd/issues/16028).

Every write to etcd bumps a global revision counter, and k8s resourceVersion is derived off of it. Controllers and operators do not poll, they use informers, which are watch-based local caches that track "I have seen up to RV X" and get notified on changes.

Here is the problem - a restore rewinds etcd's revision to the snapshot's value, which is lower than the RVs clients were already holding before the disaster. New writes then count up from that lower number, so fresh changes get RVs below what informers believe they have already seen. The watch caches do not refresh correctly, and controllers quietly act on stale state or miss post-restore changes entirely.

etcd added two `etcdutl snapshot restore` flags specifically for this, and they must be used [together](https://github.com/etcd-io/etcd/issues/16160):

*   `--bump-revision <n>`, add n to the restored keyspace's revision so revisions never move backward.
    
*   `--mark-compacted`, **mark that bumped revision as the compaction point, which terminates existing watches and makes etcd refuse to serve revisions after the snapshot, forcing every informer to re-list cleanly.** It is required when `--bump-revision > 0`, and disallowed otherwise.
    

A common sizing, straight from the etcd docs, is to bump by 1,000,000,000, which covers a week-old snapshot at under 1500 writes per second.

```bash
etcdutl snapshot restore /root/etcd-snapshot.db \
  --name master1-<cluster> \
  --initial-cluster master1-<cluster>=https://10.0.0.1:2380 \
  --initial-advertise-peer-urls https://10.0.0.1:2380 \
  --data-dir /var/lib/etcd/.restore-tmp \
  --bump-revision 1000000000 --mark-compacted
```

**What it looks like when you skip it.** This is the part the docs do not cover. The cluster comes up, `kubectl get` returns correct data, so it looks healthy. But watch-driven components misbehave:

*   Controllers and operators do not react to changes made after the restore. You edit a Deployment or a custom resource (CR) and nothing reconciles, no new ReplicaSet, no operator action.
    
*   Component logs fill with "too old resource version", "the server has received a request that reused a resource version", or repeated re-list loops.
    
*   Leader election, which rides on RVs in Lease objects, can flap.
    

**How to diagnose it.** The tell is that reads work but watches do not. So:

*   Check etcd's current revision, `etcdctl endpoint status -w json`, and notice it is far below where the cluster was before the restore.
    
*   Grep kube-controller-manager, kube-scheduler, and operator logs for "too old resource version" and re-list churn.
    
*   Drop a marker, create a ConfigMap or bump an annotation, and watch whether its consumer notices. If a known-good reconcile does not fire while `kubectl get` shows the object fine, your informers are stale.
    

**If you already restored without the flags:** the only practical fix is to force every watch to re-list, restart kube-controller-manager, kube-scheduler, and any operators, and restart kubelets for the data plane. Doing the restore with `--bump-revision --mark-compacted` in the first place is far cleaner, because it invalidates the caches for you.

Think of it as the CP counterpart to the kubelet restart below. bump-revision fixes informer staleness for controllers and operators, kubelet restart fixes it for the node agents. You want both.

## After the restore, reconcile the data plane

A restore rewinds the API state, but not the workloads actually running on nodes, and every kubelet's cached watch is now stale because etcd's revision jumped backward. So kubelets log errors and Pods drift from what etcd believes. In my tests the fix is a rolling `systemctl restart kubelet` across every node, CP and workers, one at a time, each gated on the node returning Ready, and for masters gated on etcd staying healthy between restarts. A kubelet restart does not kill running containers, it just forces kubelet to re-list, re-watch, and reconcile.

## Confirm you actually recovered

Do not declare victory on "the Pods look up". Check the things that prove the cluster is genuinely whole:

*   `etcdctl endpoint status -w table` and `endpoint health` across all three endpoints, expect three members, one leader, all healthy, `IS LEARNER=false`.
    
*   `kubectl get nodes`, every node Ready.
    
*   The marker test from the revision section, a fresh object actually gets reconciled, which proves watches are live, not just reads.
    

That last check is the one people skip, and it is the one that catches the revision-bump problem before your users do.

## TLDR: The Summary

*   Triage first. Only all-3-down is a restore.
    
*   Restore once on master1, then `member add` master2 and master3, one at a time, each gated on cluster health.
    
*   Respect the mount, match the member name, force `state=existing`, never bounce a member without quorum, and never boot stale data.
    
*   Restore with `--bump-revision --mark-compacted` so controllers' informers do not go stale on the rewound revision.
    
*   Restart kubelet fleet-wide afterward to resync the data plane.
    
*   Confirm with member health plus a live reconcile test, not just `kubectl get`.
    
*   Time it end to end, snapshot fetch included.
    

None of the individual pieces are exotic. **The value is in the order, the gates, and the specific ways it breaks,** which is the stuff you do not want to be discovering live at 3am with an outage bridge listening.

## Sources

*   [Operating etcd clusters for Kubernetes, kubernetes.io](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)
    
*   [Disaster recovery, etcd.io](https://etcd.io/docs/v3.5/op-guide/recovery/)
    
*   [Restoring etcd quorum, Red Hat OpenShift docs](https://docs.redhat.com/en/documentation/openshift_container_platform/3.11/html/cluster_administration/assembly_restore-etcd-quorum)
    
*   [Restoring a kubeadm Kubernetes Cluster from an etcd Backup, Medium](https://medium.com/@tradingcontentdrive/restoring-a-kubeadm-kubernetes-cluster-from-an-etcd-backup-aec18ece152d)
    
*   [ocp42-etcd-backup-restore-ansible, GitHub](https://github.com/acidonper/ocp42-etcd-backup-restore-ansible)
    
*   [Restoring etcd without breaking API guarantees (aka etcd restore for Kubernetes), etcd issue #16028](https://github.com/etcd-io/etcd/issues/16028)
    
*   [etcdutl README (snapshot restore flags), etcd-io/etcd](https://github.com/etcd-io/etcd/blob/main/etcdutl/README.md)