We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
An SRE at a mid-sized company spends Monday morning manually rotating TLS certificates for 12 services. Tuesday she restarts a flapping pod because the OOM killer hit it again. Wednesday she runs a database migration script by hand because the automation was never finished. Thursday she copies the same log query into three dashboards for a capacity report. Friday she does it all again.
None of that work made the system better. None of it will prevent the same tasks from showing up next week. None of it required her decade of distributed systems experience. This is toil, and it is the single biggest threat to SRE team effectiveness.
Google SRE defined the term precisely in the original SRE book: toil is work that is manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with service growth. The precision matters. A design review is not toil because it creates enduring value. Debugging a novel incident is not toil because it is not repetitive. Rotating certificates by hand every 90 days is textbook toil because it checks every box.
The Google mandate is that no SRE team should spend more than 50% of their time on toil. When a team crosses the cap, the service is either handed back to the development team or a funded project automates the top source. Etsy went from 60% toil to under 20% over one year by following this exact loop.
Most teams cannot tell you how many hours per week they spend on toil, let alone name the top five sources. Without measurement, toil reduction becomes a gut-feel project that loses to whatever feature is on the roadmap.
| Toil Criterion | Test Question | Example |
|---|---|---|
| Manual | Does a human have to do it? | SSH into a box and run a script |
| Repetitive | Does it happen more than once? | Cert renewal every 90 days |
| Automatable | Could code do it with current tools? | Restarting a pod after OOM |
| Tactical | Is it a reaction, not a planned project? | Responding to a ticket at 2am |
| No enduring value | Does the system improve after? | No, same task next quarter |
| Linear scaling | Does more traffic mean more of it? | Manual shard rebalancing |
If a task passes all six criteria, it is toil. If it passes four or five, it is probably toil. If it passes two or three, investigate further before investing in automation.
The measurement technique is tagging. Every ticket and runbook execution gets a toil tag at close time. After three weeks the data is enough to rank sources by hours consumed and justify engineering investment. Spotify publishes a toil board reviewed every sprint where each item carries a weekly hour cost, an owner, and a target automation state.