Summary - Monitoring

Combine Prometheus and Grafana to write a metrics query.

Summary: monitoring

In this group:

Prometheus pulls the targets' /metrics; types: counter, gauge, histogram, summary.
kube-prometheus-stack installs Prometheus, Alertmanager and Grafana; a ServiceMonitor is a target definition (mind the release label).
Grafana dashboards follow the RED (Rate, Errors, Duration) and USE methods.
PromQL: label filters, rate(), increase(), sum by, histogram_quantile.
A PrometheusRule holds alerting rules (expr, for, severity); Alertmanager groups and routes.

sum by (path) (rate(http_requests_total{status=~"5.."}[5m]))

Exercise Add a metrics port to the service

Write a Service YAML: name api, label app: api; selector app: api; a single port: name: http, port: 8000, targetPort: 8000. (The ServiceMonitor uses this named port.)

Show example solution

apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app: api
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 8000
      targetPort: 8000

Test yourself

A new ServiceMonitor's target doesn't appear on Prometheus's Targets page. What is checked first?
- The PVC size
- Whether the release label and the Service selector match
- The Grafana password
- The node count
Show answer

Whether the release label and the Service selector match — The Prometheus object's serviceMonitorSelector must select the ServiceMonitor.

Calculate the 95th percentile.

____(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Show answer

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Match.

Prometheus

Collects
Grafana

Displays
Alertmanager

Notifies
ServiceMonitor

Defines the target

Next lesson: Centralized logging

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies