Backup strategy

Learn cluster backup strategies.

Backup strategy

To recover a cluster from disaster you need to know what is backed up. There are three layers:

Cluster state: all Kubernetes objects; it lives in etcd.
Persistent data: the database and files on PersistentVolumes; it is not in etcd.
Definitions: the manifests, charts and values files in Git (already versioned with GitOps).

The 3-2-1 rule: 3 copies of the data, on 2 different media, 1 of them elsewhere (another region or account).

etcd snapshot     -> s3://yedek-eu/etcd/      (every hour)
Velero backup     -> s3://yedek-tr/velero/    (every night, keep 30 days)
Git repository    -> GitHub + mirror          (every commit)

Testing the backup

A backup that has never been restored doesn't count as a backup. A restore drill to an empty cluster is done regularly and the duration is measured. Encryption and retention of backups are also tied to policy:

# every month: drill
1. set up the staging-dr cluster
2. restore the latest Velero backup
3. run the application tests
4. record the duration (target: under 1 hour)

Exercise List the backups

Write three commands: list the files under s3://yedek-eu/etcd/ (aws s3 ls); list the PersistentVolumeClaims in the cluster across all namespaces (-A); list the Velero backups (velero backup get).

Show example solution

aws s3 ls s3://yedek-eu/etcd/
kubectl get pvc -A
velero backup get

Test yourself

In a cluster where only etcd is backed up, what is lost?
- Deployment definitions
- ConfigMaps
- Secrets
- The database data on PersistentVolumes
Show answer

The database data on PersistentVolumes — etcd holds the objects; data on disks must be backed up separately.

The number of copies kept elsewhere in the 3-2-1 rule.

3 copies, 2 media, ____ of them in another region

Show answer

3 copies, 2 media, 1 of them in another region

Match.

etcd

Cluster objects
PersistentVolume

Application data
Git

Definitions
tatbikat

Restore trial

Next lesson: etcd backup

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies