We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
Manual, repetitive, automatable operational work that scales linearly with service size. SRE teams aim to keep toil below 50% of their time and automate the rest.
In short
Toil is manual, repetitive operational work that is tied to running a production service, has no lasting value, and scales linearly with the size of the service. Google's SRE practice treats toil as something to measure and cap, with teams aiming to keep it under 50 percent of their time and automate the rest into engineering work.
In short
Toil is manual, repetitive operational work that is tied to running a production service, has no lasting value, and scales linearly with the size of the service. Google's SRE practice treats toil as something to measure and cap, with teams aiming to keep it under 50 percent of their time and automate the rest into engineering work.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Toil across a run of lessons, not one page. These 4 alone are about 130 minutes of step-by-step reading, every one with a quiz.
Eliminate repetitive manual work through automation, the SRE mission to reduce operational burden.
Define and measure Service Level Objectives, the practice that aligns reliability with business needs.
Real-time collaborative editing, how Google Docs lets multiple people type simultaneously.
Design a web search engine - crawling, inverted indexes, PageRank, query processing, and caching at global scale.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Toil Reductionpay once,
yours for life
Toil is the grunt work of keeping a service alive. The classic definition from Google's Site Reliability Engineering book lists specific traits: the work is manual, repetitive, automatable, reactive rather than proactive, has no enduring value, and grows in proportion to traffic or service size. If a task hits most of those marks, it is toil.
Concrete examples are easy to spot once you know the shape. Manually restarting a stuck process every night. Approving the same quota request by hand ten times a week. Applying a config change across 200 servers one SSH session at a time. Acknowledging a pager alert, running a known fix, and closing the ticket, then doing it again tomorrow because nothing changed.
The important contrast is with overhead and with real engineering. Overhead is things like email, meetings, and expense reports, which are not tied to running a service. Engineering work produces something permanent: a script, a feature, a self-healing system that makes the toil go away for good. Toil sits in between, and the danger is that it quietly fills your day while leaving nothing behind.
Toil scales with the service. A team that handles requests by hand can serve a fixed amount of load, so as traffic doubles the team has to double too. That is the opposite of what you want from an engineering organization. The fix is automation, but you only get time to automate if you protect time away from the toil itself.
Google's rule is that SREs should spend at most 50 percent of their time on toil, and ideally less. The other half goes to engineering that reduces future toil: writing tooling, hardening systems, removing the root cause of recurring alerts. When a team blows past the 50 percent ceiling for too long, that is a signal to push work back to the development team, hire, or reduce the operational surface.
Measuring toil is part of the discipline. Teams track it through ticket counts, on-call interrupt logs, and time surveys. You cannot manage what you do not count, and a number on a dashboard is what turns a vague feeling of being swamped into a budget conversation with management.
Not all toil should be eliminated immediately. The honest calculation is whether the engineering cost to automate a task is less than the toil it saves over time. If a task takes five minutes once a quarter, spending two weeks automating it is a bad trade. The phrase people use is that automation has to pay for itself.
There is also a risk in over-automating. A script that automatically fixes a problem can hide a worsening underlying issue, or it can act on a situation it was never designed for and cause a wider outage. Good automation includes guardrails, dry-run modes, and clear failure behavior rather than blindly doing the thing a human used to do.
The other trap is treating toil reduction as a one-time project. Services change, new features add new manual steps, and toil regrows like weeds. Keeping it under control is an ongoing practice, which is why it shows up in quarterly planning and on-call retrospectives rather than a single cleanup sprint.
Imagine an on-call engineer who gets paged three times a week because a service runs out of disk space. Each page, they log in, delete old log files, and restart the writer. That is textbook toil: manual, repetitive, reactive, automatable, and it grows as more instances are added.
The toil-reducing move is not to get faster at deleting logs. It is to fix the cause: add log rotation, set a retention policy, alert on the trend before disk fills, and have the system reclaim space on its own. After the work lands, the three weekly pages drop to zero, the engineer's interrupt time falls, and the fix keeps paying off as the fleet grows.
That arc is the whole point of tracking toil. The first version of the job was an endless treadmill that scaled with the service. The engineered version made the problem disappear and freed the team to work on the next source of pain instead of running in place.
Google SRE
Coined the modern definition of toil and the 50 percent cap, popularized through the Site Reliability Engineering book and its workbook.
Kubernetes
Reduces operational toil through self-healing controllers that restart, reschedule, and scale workloads automatically instead of by hand.
PagerDuty
Surfaces on-call interrupt data so teams can quantify recurring alerts and target the most toilsome ones for automation.
Netflix
Invests heavily in self-service platforms and chaos tooling so engineers automate operational tasks rather than running them manually.
No. Technical debt is shortcuts in code or design that make future changes harder. Toil is the manual operational work of running the service day to day. They are related because messy systems often create more toil, but you can have heavy toil on perfectly clean code, and clean code with little toil.
Some toil is cheaper to live with than to automate, and chasing zero would mean automating tasks that almost never happen. The 50 percent ceiling guarantees SREs keep enough time for engineering that prevents future toil, while accepting that a residual amount is normal and acceptable.
No. That is classified as overhead, not toil. Toil is specifically tied to running a production service and grows with the service. Administrative work is separate and is tracked differently in SRE time accounting.
Common methods are counting tickets and on-call pages, tagging interrupts in the issue tracker, and running periodic time surveys where engineers report what fraction of their week went to manual operations. The resulting percentage is reviewed in planning to decide where to invest in automation.
Primarily the SRE or platform team that feels it, but reduction often requires pushing work back to the developers who own the service. When toil exceeds the budget, that becomes a management conversation about staffing, automation investment, or handing operational duties back to the development team.
SRE
Google's discipline for running reliable production systems. Applies software engineering to operations: automation over toil, SLOs over uptime promises, and error budgets for velocity.
CI/CD
Continuous Integration and Continuous Deployment: automating the process of testing and deploying code. Push code, tests run, and it ships to production automatically.
Infrastructure as Code
Managing servers, networks, and cloud resources through declarative configuration files instead of manual setup. Terraform, Pulumi, and CloudFormation are IaC tools.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The "Toil Reduction" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Toil Reduction lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Toil in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
