Toil

Toil is the grunt work of keeping a service alive. The classic definition from Google's Site Reliability Engineering book lists specific traits: the work is manual, repetitive, automatable, reactive rather than proactive, has no enduring value, and grows in proportion to traffic or service size. If a task hits most of those marks, it is toil.

Concrete examples are easy to spot once you know the shape. Manually restarting a stuck process every night. Approving the same quota request by hand ten times a week. Applying a config change across 200 servers one SSH session at a time. Acknowledging a pager alert, running a known fix, and closing the ticket, then doing it again tomorrow because nothing changed.

The important contrast is with overhead and with real engineering. Overhead is things like email, meetings, and expense reports, which are not tied to running a service. Engineering work produces something permanent: a script, a feature, a self-healing system that makes the toil go away for good. Toil sits in between, and the danger is that it quietly fills your day while leaving nothing behind.

Toil scales with the service. A team that handles requests by hand can serve a fixed amount of load, so as traffic doubles the team has to double too. That is the opposite of what you want from an engineering organization. The fix is automation, but you only get time to automate if you protect time away from the toil itself.

Google's rule is that SREs should spend at most 50 percent of their time on toil, and ideally less. The other half goes to engineering that reduces future toil: writing tooling, hardening systems, removing the root cause of recurring alerts. When a team blows past the 50 percent ceiling for too long, that is a signal to push work back to the development team, hire, or reduce the operational surface.

Measuring toil is part of the discipline. Teams track it through ticket counts, on-call interrupt logs, and time surveys. You cannot manage what you do not count, and a number on a dashboard is what turns a vague feeling of being swamped into a budget conversation with management.

Not all toil should be eliminated immediately. The honest calculation is whether the engineering cost to automate a task is less than the toil it saves over time. If a task takes five minutes once a quarter, spending two weeks automating it is a bad trade. The phrase people use is that automation has to pay for itself.

There is also a risk in over-automating. A script that automatically fixes a problem can hide a worsening underlying issue, or it can act on a situation it was never designed for and cause a wider outage. Good automation includes guardrails, dry-run modes, and clear failure behavior rather than blindly doing the thing a human used to do.

The other trap is treating toil reduction as a one-time project. Services change, new features add new manual steps, and toil regrows like weeds. Keeping it under control is an ongoing practice, which is why it shows up in quarterly planning and on-call retrospectives rather than a single cleanup sprint.

Imagine an on-call engineer who gets paged three times a week because a service runs out of disk space. Each page, they log in, delete old log files, and restart the writer. That is textbook toil: manual, repetitive, reactive, automatable, and it grows as more instances are added.

The toil-reducing move is not to get faster at deleting logs. It is to fix the cause: add log rotation, set a retention policy, alert on the trend before disk fills, and have the system reclaim space on its own. After the work lands, the three weekly pages drop to zero, the engineer's interrupt time falls, and the fix keeps paying off as the fleet grows.

That arc is the whole point of tracking toil. The first version of the job was an endless treadmill that scaled with the service. The engineered version made the problem disappear and freed the team to work on the next source of pain instead of running in place.