Self-healing

Learn that ReplicaSet recreates deleted Pods.

Self-healing

The ReplicaSet controller runs a continuous reconciliation loop: it observes the actual pod count, compares it with the desired one, and closes the gap. If a pod is deleted, a node crashes or a pod is evicted, the missing pod is recreated:

$ kubectl delete pod web-7c5b9f6d8-abcde
$ kubectl get events --field-selector involvedObject.kind=ReplicaSet | tail -2
Normal  SuccessfulDelete  replicaset/web-7c5b9f6d8  Deleted pod: web-7c5b9f6d8-abcde
Normal  SuccessfulCreate  replicaset/web-7c5b9f6d8  Created pod: web-7c5b9f6d8-mnopq

Container restarts

There are two levels of healing. If a container crashes, the kubelet restarts the container inside the same pod (restartPolicy: Always) and the RESTARTS counter goes up. If the pod itself disappears, the ReplicaSet creates a new pod. The log of the previously crashed container is seen with --previous:

$ kubectl get pod api-5d8f7c9b4-qrstu
NAME                  READY   STATUS    RESTARTS      AGE
api-5d8f7c9b4-qrstu   1/1     Running   3 (2m ago)    20m
$ kubectl logs api-5d8f7c9b4-qrstu --previous

Exercise Observe the healing

Write three commands: delete the web-7c5b9f6d8-abcde pod; list the pods labelled app=web; show the previous container log of the api-5d8f7c9b4-qrstu pod.

Show example solution

kubectl delete pod web-7c5b9f6d8-abcde
kubectl get pods -l app=web
kubectl logs api-5d8f7c9b4-qrstu --previous

Test yourself

What does self-healing not cover?
- A deleted pod
- A pod on a crashed node
- An evicted pod
- A bug in the application; the new pod crashes with the same bug
Show answer

A bug in the application; the new pod crashes with the same bug — A constantly crashing pod falls into CrashLoopBackOff.

Show the log of the previous (crashed) container.

kubectl logs api-5d8f7c9b4-qrstu ____

Show answer

kubectl logs api-5d8f7c9b4-qrstu --previous

Match.

kubelet

Restarts the container
ReplicaSet

Recreates the pod
RESTARTS

Container restart count
--previous

Log of the crashed one

Next lesson: Summary - ReplicaSet

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies