Fluentd

In a typical infrastructure, logs come from everywhere: application stdout, syslog, Docker containers, Kubernetes pods, cloud provider APIs, message queues, load balancers. Each source outputs in a different format. Each destination (Elasticsearch, S3, Datadog, Splunk) expects a different input format.

Without a unified log collector, you end up with a mess of custom scripts and fragile pipelines. Fluentd was built to solve this: a single daemon that accepts logs from any source (800+ input plugins), transforms them (400+ filter plugins), and routes them to any destination (500+ output plugins).

Fluentd graduated from the CNCF in 2019 and is the de facto standard for log collection in Kubernetes environments. The Kubernetes logging architecture documentation recommends Fluentd (or its lighter variant, Fluent Bit) as the cluster-level logging agent.

Fluentd's architecture is a pipeline: Input → Filter → Output

Input plugins collect data: tail (follow log files), forward (receive from other Fluentd instances), http (accept HTTP POSTs), syslog, docker, and hundreds more.

Filter plugins transform data: parser (parse unstructured text into fields), record_transformer (add/modify fields), grep (include/exclude records), concat (merge multi-line logs).

Output plugins send data to destinations: elasticsearch, s3, kafka, datadog, splunk, loki, stdout.

Fluent Bit is Fluentd's lightweight sibling, written in C (vs Fluentd's Ruby), uses 450KB of memory (vs Fluentd's 40MB+). Fluent Bit runs on each node as a DaemonSet in Kubernetes, doing lightweight collection and forwarding. Fluentd runs as a central aggregator doing heavy processing. The pattern: Fluent Bit (edge) → Fluentd (aggregator) → Backend.