We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
Google's discipline for running reliable production systems. Applies software engineering to operations: automation over toil, SLOs over uptime promises, and error budgets for velocity.
In short
Site Reliability Engineering (SRE) is a discipline, started at Google around 2003, that runs production systems by applying software engineering practices to operations work. Instead of promising perfect uptime, SRE teams set measurable reliability targets called SLOs, track an error budget that says how much failure is allowed, and automate manual operational work (toil) so a small team can run a large system.
In short
Site Reliability Engineering (SRE) is a discipline, started at Google around 2003, that runs production systems by applying software engineering practices to operations work. Instead of promising perfect uptime, SRE teams set measurable reliability targets called SLOs, track an error budget that says how much failure is allowed, and automate manual operational work (toil) so a small team can run a large system.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers SRE across a run of lessons, not one page. These 4 alone are about 107 minutes of step-by-step reading, every one with a quiz.
Eliminate repetitive manual work through automation, the SRE mission to reduce operational burden.
The quantified amount of unreliability you're allowed, the most powerful tool in SRE.
Define and measure Service Level Objectives, the practice that aligns reliability with business needs.
Set limits on latency, error rate, and resource usage. SLOs as engineering constraints.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Toil Reductionpay once,
yours for life
SRE is the answer to a simple question: who runs the software after engineers write it, and how do they keep it reliable without burning out? The traditional answer was a separate operations team that manually deployed, watched dashboards, and got paged at 3 AM. SRE replaces that with engineers who treat operations as a software problem. If a task is repetitive and manual, an SRE writes code to make it go away.
The term and the practice come from Google, where Ben Treynor Sloss started building the first SRE team in 2003. The core idea he describes is taking the work a sysadmin team would do and handing it to engineers who would get bored doing it by hand, so they automate it instead. Google later wrote the practice down in the 2016 book Site Reliability Engineering, which is why the field has such a consistent vocabulary.
An SRE team owns reliability as an explicit product feature. They are measured on whether the service meets its reliability target, not on whether they followed a runbook. That shift in incentive is what separates SRE from a renamed ops team.
The center of SRE is the Service Level Objective (SLO). An SLO is a number you commit to, such as 99.9 percent of requests succeed in under 300 ms, measured over 28 days. It is built from Service Level Indicators (SLIs), which are the raw measurements like success rate and latency. The SLO is what the business and the engineers agree is good enough.
Once you have an SLO, you get an error budget for free. If the SLO is 99.9 percent, the budget is the remaining 0.1 percent of allowed failure. Over a 30-day month that is about 43 minutes of downtime. The budget turns reliability into a currency. If the service is well inside its budget, the team can ship features fast and take risks. If the budget is spent, feature launches stop and everyone focuses on reliability until the budget refills. This kills the old fight between developers who want to ship and ops who want stability, because the number decides.
The other pillar is fighting toil. Toil is manual, repetitive operational work that scales linearly with traffic and produces no lasting value, like restarting a stuck server by hand or running the same deploy steps every release. Google's published guideline is that SREs should spend no more than 50 percent of their time on toil, with the rest going to engineering that reduces future toil. The team also runs blameless postmortems after incidents, focusing on the system that allowed the failure rather than punishing a person.
SRE pays off when you run a service at scale where downtime has real cost and the system is too large to babysit by hand. A consumer app with millions of users, a payments platform, or a multi-region API are all good fits. Below a certain size the full SRE model is overhead. A three-person startup does not need an error budget policy when the founders are also the on-call rotation.
The biggest trade-off is that real SRE requires organizational buy-in, not just a job title. If management overrides the error budget and forces launches when the budget is blown, the model collapses into ops with extra steps. The discipline only works if the SLO actually has teeth and can halt feature work.
There is also a cost in headcount and skill. SREs are software engineers who also understand systems, networking, and operations, which makes them hard to hire. The 100 percent reliability trap is another mistake: chasing extra nines past what users notice is expensive and often pointless. If users cannot tell the difference between 99.99 and 99.999 percent, the extra nine is wasted money.
Picture a checkout service for an online store. The team sets an SLO of 99.95 percent of checkout requests succeeding over 28 days. That gives an error budget of 0.05 percent, roughly 20 minutes of failed checkouts per month.
For the first three weeks the service is healthy and the budget is barely touched, so the team confidently ships a risky new payment provider integration behind a feature flag. A bad deploy then causes a spike of 500 errors that burns through 80 percent of the remaining budget in one afternoon. Automated burn-rate alerts fire, the on-call SRE rolls back, and a blameless postmortem produces an action item to add a canary deploy step.
Because the budget is now nearly spent, the team freezes new feature launches and spends the rest of the month hardening the deploy pipeline and adding the canary check. Next month the budget resets, the system is more reliable than before, and feature work resumes. Nobody argued about whether to slow down because the number made the call.
Invented SRE in 2003 and runs services like Search, Gmail, and Ads on SLOs and error budgets; its 2016 book defined the field.
Amazon Web Services
Operates services to published SLAs backed by internal SLOs, with on-call ownership and automated remediation across its global regions.
Netflix
Runs a reliability-first culture with chaos engineering tools like Chaos Monkey to test that systems stay within their reliability targets.
Spotify
Embeds SRE practices across squads, using SLOs and error budgets to balance fast feature delivery against backend stability.
DevOps is a broad culture and set of principles about breaking down the wall between development and operations. SRE is one concrete, opinionated way to implement those principles, with specific practices like SLOs, error budgets, and a 50 percent cap on toil. A common saying is that SRE is what you get when you treat operations as a software problem, so SRE is a specific implementation of DevOps.
An error budget is the amount of unreliability you are allowed before you breach your SLO. If your SLO is 99.9 percent success, your error budget is the remaining 0.1 percent of requests, which over 30 days is about 43 minutes. Teams spend it on risk: when budget remains they ship fast, and when it is exhausted they freeze launches and focus on reliability.
An SLI is the raw measurement, such as the percentage of requests served in under 300 ms. An SLO is the internal target you set on that SLI, like 99.9 percent over 28 days. An SLA is a contract with customers that usually promises something looser than the SLO and includes penalties such as refunds if it is breached. SLOs are typically stricter than SLAs so you catch problems before the contract is broken.
Toil is manual, repetitive operational work that has no lasting value and grows as the service grows, like manually restarting servers or copy-pasting the same deploy commands. SRE treats toil as something to automate away, and Google's guideline caps SRE time spent on toil at 50 percent so the rest goes toward engineering that eliminates future toil.
Not the full model. The SRE discipline pays off at scale where downtime is costly and a system is too large to run by hand. A small startup is better off adopting a few cheap pieces, like defining one or two SLOs and writing blameless postmortems, without a dedicated SRE team or a formal error budget policy.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.
Error Budget
The allowed amount of unreliability derived from SLOs. If your SLO is 99.9% uptime, your error budget is 0.1% (about 43 minutes/month). Once exhausted, freeze deployments.
Postmortem
A blameless analysis conducted after an incident to document what happened, why, and how to prevent it from recurring. The most important output is the list of action items.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The course covers it step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
See the System Design Masterclass coursecourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about SRE in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
