Vector

A fintech company was processing 50 GB of logs per hour through Fluentd. The pipeline was consuming significant CPU and memory, adding noticeable latency, and occasionally dropping logs during traffic spikes when the buffer filled up. They needed something faster.

Vector, built by Timber (now Datadog, which acquired the project), is written in Rust, a language designed for performance and memory safety. It handles the same log/metrics pipeline workload as Fluentd or Logstash but typically uses 2-10x less CPU and memory. For high-throughput environments, this performance advantage translates directly into fewer pipeline instances and lower infrastructure costs.

Vector handles both logs AND metrics in a single pipeline. While Fluentd is primarily a log processor and Prometheus handles metrics separately, Vector can collect, transform, and route both data types through one unified configuration.

Vector uses the same source → transform → sink pipeline model:

VRL (Vector Remap Language) is Vector's data transformation language, purpose-built for observability data manipulation. It's type-safe, performant, and designed to be easier than Logstash's grok patterns or Fluentd's record_transformer.

# Parse a log line and add fields
. = parse_syslog!(.message)
.env = "production"
.region = get_env_var!("AWS_REGION")
del(.hostname) # Remove sensitive field