We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Observability & Monitoring
An open-source monitoring system that scrapes metrics endpoints, stores time-series data, and supports powerful PromQL queries. The de facto standard for Kubernetes monitoring.
In short
Prometheus is an open-source monitoring system that pulls numeric metrics from your applications and infrastructure on a fixed interval, stores them as time-series data labeled by dimensions like instance and route, and lets you query and alert on that data using its own query language, PromQL. It is the default monitoring stack for Kubernetes and graduated from the Cloud Native Computing Foundation in 2018.
In short
Prometheus is an open-source monitoring system that pulls numeric metrics from your applications and infrastructure on a fixed interval, stores them as time-series data labeled by dimensions like instance and route, and lets you query and alert on that data using its own query language, PromQL. It is the default monitoring stack for Kubernetes and graduated from the Cloud Native Computing Foundation in 2018.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Prometheus across a run of lessons, not one page. These 3 alone are about 43 minutes of step-by-step reading, every one with a quiz.
The open-source metrics system that became the industry standard, pull-based collection, powerful queries, and native Kubernetes integration.
Databases built for timestamped data, metrics, IoT sensors, financial ticks, and observability.
Monotonically increasing numbers that count events, requests served, errors thrown, bytes transferred.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Prometheuspay once,
yours for life
Prometheus is a single Go binary that collects metrics, stores them, and answers queries. It was built at SoundCloud in 2012, inspired by Google's internal Borgmon system, and was the second project to join the CNCF after Kubernetes itself. That history is why the two are so tightly paired.
A metric in Prometheus is a number that changes over time, tagged with key-value labels. For example, http_requests_total{method="POST", route="/checkout", status="500"} is one time series. The same metric name with different label values is a different series. This label model is the core idea: instead of one flat counter, you get a multi-dimensional cube you can slice any way you want at query time.
Everything is stored as a float64 sample plus a millisecond timestamp. There are four metric types: counters that only go up, gauges that go up and down, histograms that bucket observations like request latency, and summaries that compute quantiles client-side.
Prometheus is pull-based, not push-based. On a schedule (commonly every 15 to 30 seconds) the server scrapes an HTTP endpoint, usually /metrics, on every target. The target exposes its current metric values as plain text. This is the opposite of systems like StatsD where the application pushes data out.
Targets are found through service discovery. In Kubernetes, Prometheus watches the API server and automatically discovers pods and services as they come and go, so you do not hand-edit a target list. It also supports EC2, Consul, DNS, and static configs.
Samples are written to a local time-series database on disk. Recent data lives in an in-memory head block and is flushed to immutable two-hour blocks, which are later compacted into larger blocks. A write-ahead log protects against crashes. By default retention is 15 days because Prometheus is designed for recent operational data, not long-term archival.
Alerting is split out. Prometheus evaluates alerting rules written in PromQL and, when one fires, sends it to a separate component called Alertmanager, which handles grouping, deduplication, silencing, and routing to Slack, PagerDuty, or email.
Reach for Prometheus when you need operational metrics: request rates, error rates, latency percentiles, CPU and memory, queue depth. PromQL makes ratios and rates easy. rate(http_requests_total[5m]) gives per-second request rate, and dividing the error series by the total series gives an error ratio you can alert on.
The main limitation is that a single Prometheus server is not horizontally scalable and not built for long-term storage or global views across many clusters. When you outgrow one server you reach for Thanos, Cortex, or Grafana Mimir, which add object-storage backends, deduplication, and a unified query layer over many Prometheus instances.
Prometheus is also for metrics only. It is not a logging system and not a distributed tracing system. Pull-based scraping struggles with short-lived batch jobs that finish before a scrape happens, which is why the Pushgateway exists as a workaround for that specific case. High label cardinality, like putting a unique user ID or full URL into a label, can blow up memory and is the most common way people break a Prometheus server.
Say you run a payments API on Kubernetes. You add a client library (the official ones cover Go, Java, Python, and more) that exposes /metrics, including a histogram http_request_duration_seconds and a counter http_requests_total.
Prometheus discovers your pods automatically and scrapes them every 15 seconds. In Grafana you build a dashboard that queries histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) to show p99 latency, which is the number your SLO is written against.
You write an alerting rule: if the 5-minute error ratio stays above 1 percent for 10 minutes, fire HighErrorRate. Prometheus evaluates that rule, hands the firing alert to Alertmanager, and Alertmanager pages the on-call engineer in PagerDuty while suppressing duplicates. This metrics-plus-Grafana-plus-Alertmanager stack is the standard observability setup in most cloud-native shops today.
Kubernetes
kube-state-metrics and node-exporter expose cluster and node metrics in Prometheus format, and the kube-prometheus-stack Helm chart is the default way to monitor a cluster.
SoundCloud
Built Prometheus internally in 2012 to monitor its microservices after outgrowing earlier tooling, then open-sourced it.
Grafana Labs
Grafana is the most common dashboard front end for Prometheus, and Grafana Mimir scales Prometheus storage to billions of active series.
CloudFlare
Runs large Prometheus deployments to monitor its global edge network and has published its own tooling and patterns for high-cardinality metrics.
Pulling lets Prometheus control the scrape rate, easily tell when a target is down (a failed scrape is itself a signal), and discover targets dynamically instead of having every service know where to send data. Push-based systems shift that burden onto the application and make a missing service look the same as a healthy quiet one.
Prometheus collects, stores, and queries metrics and handles alerting rules. Grafana is a visualization tool that reads from Prometheus (and many other sources) to draw dashboards. They are complementary, not competitors, which is why they are almost always run together.
Not well on its own. Default retention is 15 days and a single server keeps data on local disk. For long-term, durable, or cross-cluster storage you front Prometheus with Thanos, Cortex, or Grafana Mimir, which offload blocks to object storage like S3 and add a global query view.
PromQL is Prometheus's query language for time-series data. It supports selecting series by metric name and labels, computing rates over counters with rate(), aggregating with sum and avg by label, and deriving percentiles from histograms. Both dashboards and alerting rules are written in PromQL.
Cardinality is the number of unique label combinations for a metric. Each combination is its own time series held in memory. Putting unbounded values like user IDs, email addresses, or full request URLs into labels creates millions of series and is the most common cause of a Prometheus server running out of memory and crashing.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
Alerting
Automatically notifying engineers when metrics cross predefined thresholds. Good alerts are actionable, not noisy. PagerDuty and Opsgenie route alerts to the right on-call person.
Kubernetes
An orchestration platform that automates deploying, scaling, and managing containerized applications. K8s is the operating system for your cloud.
Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
This page explains the idea. The "Prometheus" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Prometheus lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Prometheus in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
