Prometheus
An open-source monitoring system that scrapes metrics endpoints, stores time-series data, and supports powerful PromQL queries. The de facto standard for Kubernetes monitoring.
What is Prometheus?
In short
Prometheus is an open-source monitoring system that pulls numeric metrics from your applications and infrastructure on a fixed interval, stores them as time-series data labeled by dimensions like instance and route, and lets you query and alert on that data using its own query language, PromQL. It is the default monitoring stack for Kubernetes and graduated from the Cloud Native Computing Foundation in 2018.
What Prometheus actually is
Prometheus is a single Go binary that collects metrics, stores them, and answers queries. It was built at SoundCloud in 2012, inspired by Google's internal Borgmon system, and was the second project to join the CNCF after Kubernetes itself. That history is why the two are so tightly paired.
A metric in Prometheus is a number that changes over time, tagged with key-value labels. For example, http_requests_total{method="POST", route="/checkout", status="500"} is one time series. The same metric name with different label values is a different series. This label model is the core idea: instead of one flat counter, you get a multi-dimensional cube you can slice any way you want at query time.
Everything is stored as a float64 sample plus a millisecond timestamp. There are four metric types: counters that only go up, gauges that go up and down, histograms that bucket observations like request latency, and summaries that compute quantiles client-side.
How it works under the hood
Prometheus is pull-based, not push-based. On a schedule (commonly every 15 to 30 seconds) the server scrapes an HTTP endpoint, usually /metrics, on every target. The target exposes its current metric values as plain text. This is the opposite of systems like StatsD where the application pushes data out.
Targets are found through service discovery. In Kubernetes, Prometheus watches the API server and automatically discovers pods and services as they come and go, so you do not hand-edit a target list. It also supports EC2, Consul, DNS, and static configs.
Samples are written to a local time-series database on disk. Recent data lives in an in-memory head block and is flushed to immutable two-hour blocks, which are later compacted into larger blocks. A write-ahead log protects against crashes. By default retention is 15 days because Prometheus is designed for recent operational data, not long-term archival.
Alerting is split out. Prometheus evaluates alerting rules written in PromQL and, when one fires, sends it to a separate component called Alertmanager, which handles grouping, deduplication, silencing, and routing to Slack, PagerDuty, or email.
When to use it and the trade-offs
Reach for Prometheus when you need operational metrics: request rates, error rates, latency percentiles, CPU and memory, queue depth. PromQL makes ratios and rates easy. rate(http_requests_total[5m]) gives per-second request rate, and dividing the error series by the total series gives an error ratio you can alert on.
The main limitation is that a single Prometheus server is not horizontally scalable and not built for long-term storage or global views across many clusters. When you outgrow one server you reach for Thanos, Cortex, or Grafana Mimir, which add object-storage backends, deduplication, and a unified query layer over many Prometheus instances.
Prometheus is also for metrics only. It is not a logging system and not a distributed tracing system. Pull-based scraping struggles with short-lived batch jobs that finish before a scrape happens, which is why the Pushgateway exists as a workaround for that specific case. High label cardinality, like putting a unique user ID or full URL into a label, can blow up memory and is the most common way people break a Prometheus server.
A concrete example
Say you run a payments API on Kubernetes. You add a client library (the official ones cover Go, Java, Python, and more) that exposes /metrics, including a histogram http_request_duration_seconds and a counter http_requests_total.
Prometheus discovers your pods automatically and scrapes them every 15 seconds. In Grafana you build a dashboard that queries histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) to show p99 latency, which is the number your SLO is written against.
You write an alerting rule: if the 5-minute error ratio stays above 1 percent for 10 minutes, fire HighErrorRate. Prometheus evaluates that rule, hands the firing alert to Alertmanager, and Alertmanager pages the on-call engineer in PagerDuty while suppressing duplicates. This metrics-plus-Grafana-plus-Alertmanager stack is the standard observability setup in most cloud-native shops today.
Where it is used in production
Kubernetes
kube-state-metrics and node-exporter expose cluster and node metrics in Prometheus format, and the kube-prometheus-stack Helm chart is the default way to monitor a cluster.
SoundCloud
Built Prometheus internally in 2012 to monitor its microservices after outgrowing earlier tooling, then open-sourced it.
Grafana Labs
Grafana is the most common dashboard front end for Prometheus, and Grafana Mimir scales Prometheus storage to billions of active series.
CloudFlare
Runs large Prometheus deployments to monitor its global edge network and has published its own tooling and patterns for high-cardinality metrics.
Frequently asked questions
- Why is Prometheus pull-based instead of push-based?
- Pulling lets Prometheus control the scrape rate, easily tell when a target is down (a failed scrape is itself a signal), and discover targets dynamically instead of having every service know where to send data. Push-based systems shift that burden onto the application and make a missing service look the same as a healthy quiet one.
- What is the difference between Prometheus and Grafana?
- Prometheus collects, stores, and queries metrics and handles alerting rules. Grafana is a visualization tool that reads from Prometheus (and many other sources) to draw dashboards. They are complementary, not competitors, which is why they are almost always run together.
- Can Prometheus store data long-term?
- Not well on its own. Default retention is 15 days and a single server keeps data on local disk. For long-term, durable, or cross-cluster storage you front Prometheus with Thanos, Cortex, or Grafana Mimir, which offload blocks to object storage like S3 and add a global query view.
- What is PromQL?
- PromQL is Prometheus's query language for time-series data. It supports selecting series by metric name and labels, computing rates over counters with rate(), aggregating with sum and avg by label, and deriving percentiles from histograms. Both dashboards and alerting rules are written in PromQL.
- What is high cardinality and why does it matter?
- Cardinality is the number of unique label combinations for a metric. Each combination is its own time series held in memory. Putting unbounded values like user IDs, email addresses, or full request URLs into labels creates millions of series and is the most common cause of a Prometheus server running out of memory and crashing.
Learn Prometheus hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Prometheus as part of a larger topic.
Time-Series Databases
Databases built for timestamped data, metrics, IoT sensors, financial ticks, and observability
intermediate · database types storage
Counter Metrics
Monotonically increasing numbers that count events, requests served, errors thrown, bytes transferred
intermediate · observability monitoring
See also
Related glossary terms you might want to look up next.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
Kubernetes
An orchestration platform that automates deploying, scaling, and managing containerized applications. K8s is the operating system for your cloud.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.