Alertmanager
- Docker & Kubernetes course
- Advanced
- Lesson 53/70
- 4 min
Learn alert management with Alertmanager.
Alertmanager
Prometheus evaluates alerting rules; if a condition holds for a given time (for), it sends the alert to Alertmanager. Alertmanager groups alerts, suppresses repeats and routes them to the right channel (email, Slack, PagerDuty). In the operator, rules are written with a PrometheusRule object:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: api-uyarilari
namespace: izleme
labels: {release: izleme}
spec:
groups:
- name: api
rules:
- alert: YuksekHataOrani
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: critical
annotations:
summary: "API hata oranı %5'in üstünde"Good alerts
Alert fatigue is when unnecessary alerts overshadow important ones. Good alerts are based on symptoms (the error rate or latency the user experiences), not causes (CPU at 80%); and every alert has a link explaining what to do (a runbook):
annotations:
summary: "API hata oranı %5'in üstünde"
runbook_url: https://wiki.sirket.com/runbook/api-hata
kubectl get prometheusrules -n izlemePractice An alerting rule
Write a PrometheusRule's spec section (starting with spec:): a group named api under groups; a single rule: alert: PodYenidenBasliyor, expr: increase(kube_pod_container_status_restarts_total[15m]) > 3, for: 5m, labels.severity: warning.
Starter code
# buraya yazSample solution
spec:
groups:
- name: api
rules:
- alert: PodYenidenBasliyor
expr: increase(kube_pod_container_status_restarts_total[15m]) > 3
for: 5m
labels:
severity: warningExpected output
prometheusrule.monitoring.coreos.com/api-uyarilari configured
Questions and answers
Review the lesson questions together with their answers. To test yourself, solve the lesson interactively.
1 What does the for: 10m field do?
- The alert is deleted after 10 minutes
- The query looks at 10 minutes of data
- The alert fires if the condition holds for 10 minutes straight; short spikes are ignored Correct answer
- The alert is sent every 10 minutes
Explanation: During that time the alert waits in the Pending state.
2 Write how long the condition must hold.
- alert: YuksekHataOrani
expr: hata_orani > 0.05
____: 10m - alert: YuksekHataOrani
expr: hata_orani > 0.05
for: 10mExplanation: for is how long the condition must hold uninterrupted.
3 Match.
| Term | Meaning |
|---|---|
PrometheusRule |
Alerting rules |
Alertmanager |
Groups and routes |
severity |
Severity label |
runbook_url |
What to do |