Docker & Kubernetes
Alertmanager
Learn alert management with Alertmanager.
Alertmanager
Prometheus evaluates alerting rules; if a condition holds for a given time (for), it sends the alert to Alertmanager. Alertmanager groups alerts, suppresses repeats and routes them to the right channel (email, Slack, PagerDuty). In the operator, rules are written with a PrometheusRule object:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: api-uyarilari
namespace: izleme
labels: {release: izleme}
spec:
groups:
- name: api
rules:
- alert: YuksekHataOrani
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: critical
annotations:
summary: "API hata oranı %5'in üstünde"
Good alerts
Alert fatigue is when unnecessary alerts overshadow important ones. Good alerts are based on symptoms (the error rate or latency the user experiences), not causes (CPU at 80%); and every alert has a link explaining what to do (a runbook):
annotations:
summary: "API hata oranı %5'in üstünde"
runbook_url: https://wiki.sirket.com/runbook/api-hata
kubectl get prometheusrules -n izleme
Exercise An alerting rule
Write a PrometheusRule's spec section (starting with spec:): a group named api under groups; a single rule: alert: PodYenidenBasliyor, expr: increase(kube_pod_container_status_restarts_total[15m]) > 3, for: 5m, labels.severity: warning.
Show example solution
spec:
groups:
- name: api
rules:
- alert: PodYenidenBasliyor
expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
for: 5m
labels:
severity: warning
Test yourself
-
What does the for: 10m field do?
- The alert is deleted after 10 minutes
- The query looks at 10 minutes of data
- The alert fires if the condition holds for 10 minutes straight; short spikes are ignored
- The alert is sent every 10 minutes
Show answer
The alert fires if the condition holds for 10 minutes straight; short spikes are ignored — During that time the alert waits in the Pending state.
-
Write how long the condition must hold.
- alert: YuksekHataOrani
expr: hata_orani > 0.05
____: 10m
Show answer
- alert: YuksekHataOrani
expr: hata_orani > 0.05
for: 10m
-
Match.
- PrometheusRule
- Alerting rules
- Alertmanager
- Groups and routes
- severity
- Severity label
- runbook_url
- What to do
Next lesson: Summary - Monitoring
