Observability and Monitoring

Monitoring answers questions you already know to ask. You decide in advance that CPU above 90 percent is bad, you set a threshold, and you get paged when it crosses. It works well for known failure modes. The problem is that distributed systems fail in ways nobody predicted, and a dashboard built for yesterday's outage tells you nothing about today's.

Observability is the broader goal: building a system whose internal state you can understand from the outside, even for failures you never anticipated. The Observability Overview lesson frames the difference, and it comes down to one test. When something breaks in a way you have never seen, can you ask new questions of your data and get answers without shipping new code? If yes, your system is observable. If you have to add logging and redeploy to understand an incident, you only have monitoring.

The practical bridge between the two is the three pillars. Logs give you detailed, timestamped records of discrete events. Metrics give you cheap, aggregatable numbers you can graph and alert on. Traces stitch a single request together as it hops across services. None of the three alone is enough. Logs without metrics drown you in detail; metrics without traces tell you that latency rose but not which service caused it.

You cannot improve what you have not defined. SLIs, SLOs, and SLAs are how teams put numbers on reliability. A Service Level Indicator is the raw measurement, like the percentage of requests served under 300 milliseconds. A Service Level Objective is the target you commit to internally, like 99.9 percent of requests under that bound over 30 days. A Service Level Agreement is the external, contractual version with financial penalties attached. The SLA/SLO/SLI Overview lesson ties the three together so you stop using the terms interchangeably.

The most useful idea that falls out of SLOs is the Error Budget. If your SLO is 99.9 percent, you are allowed to fail 0.1 percent of the time, and that allowance is a budget you can spend on risk. Plenty of budget left means ship faster. Budget exhausted means freeze features and fix reliability. It turns reliability from an argument into a number both engineers and product managers can read.

The time-based metrics quantify how well your incident response works. MTTD measures how long until you notice, MTTA how long until someone acknowledges the page, MTTR how long until it is resolved, and MTBF how long the system runs between failures. Alongside Availability Metrics, Uptime Monitoring, and Downtime Tracking, these are the numbers leadership asks about after every outage.

Theory becomes practice through tools, and this category covers the ones you will meet in real jobs. For metrics, Prometheus is the de facto open-source standard for collection and storage, almost always paired with Grafana for dashboards. StatsD and Telegraf are common metric collectors that feed into these systems.

For tracing, Jaeger and Zipkin are the leading open-source backends, and OpenTelemetry is the vendor-neutral standard that increasingly feeds all of them. OpenTelemetry matters because it lets you instrument your code once and send the data to whichever backend you choose, so you are not locked into one vendor's agent. For logs, the ELK Stack (Elasticsearch, Logstash, Kibana) plus shippers like Fluentd and Vector form the classic pipeline, while Splunk is the heavyweight enterprise option.

On the commercial side, Datadog, New Relic, AppDynamics, and Dynatrace bundle metrics, logs, and traces into one platform with less setup, at higher cost. The real-world pattern at most companies is a mix: open-source for high-volume signals where cost matters, a commercial platform where speed of setup and integrated dashboards matter more. The skill is not memorizing tools but knowing which signal each one is good at and how they fit together.