Fluentd

Learn log collection with Fluentd.

Fluentd and Fluent Bit

Fluentd and its lightweight sibling Fluent Bit are collectors that read logs (input), process them (filter/parser) and send them to destinations (output). In Kubernetes, Fluent Bit per node is common: it uses little memory and, with the kubernetes filter, adds the pod name, namespace and labels to every line:

[INPUT]
    Name   tail
    Path   /var/log/containers/*.log
    Tag    kube.*
[FILTER]
    Name   kubernetes
    Match  kube.*
    Merge_Log On
[OUTPUT]
    Name   es
    Match  *
    Host   elasticsearch.loglama.svc
    Port   9200

Installing with Helm

Fluent Bit's official Helm chart installs the DaemonSet, the RBAC permissions and the configuration together. The output destination is changed with values:

helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit -n loglama --create-namespace
kubectl get ds -n loglama
kubectl logs -n loglama ds/fluent-bit --tail=5

Exercise Install Fluent Bit

Write three commands: add the fluent repository with the address https://fluent.github.io/helm-charts; install the fluent/fluent-bit chart as fluent-bit into the loglama namespace (creating it if missing); list the DaemonSets in loglama.

Show example solution

helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit -n loglama --create-namespace
kubectl get ds -n loglama

Test yourself

What is the job of the kubernetes filter?
- Restarting pods
- Encrypting logs
- Adding pod, namespace and label information to log lines
- Deleting logs
Show answer

Adding pod, namespace and label information to log lines — It queries this info from the API server; that's why it needs pod-read RBAC permission.

Write the input plugin that reads the end of files.

[INPUT]
    Name   ____
    Path   /var/log/containers/*.log

Show answer

[INPUT]
    Name   tail
    Path   /var/log/containers/*.log

Match the sections.

[INPUT]

Reading
[FILTER]

Processing
[OUTPUT]

Sending
Match

Which tags

Next lesson: Elasticsearch

We Are Enhancing Your Experience

Cookie Preferences

Strictly Necessary Cookies

Analytical Cookies