Immutable Infrastructure

In 2012, a startup ran their entire product on a single EC2 instance that an early engineer had configured by hand two years earlier. Nobody knew every package installed on it, every config file tweaked, every cron job added during a late night debugging session. When the instance died, they spent three days trying to recreate it from memory. They lost customers. They nearly lost the company.

This is the snowflake server problem. Any server that lives long enough accumulates changes that diverge it from any documented baseline. Patches, hotfixes, one-off debugging artifacts, security tweaks applied at 3am and never recorded. After a few months, the server is unique in ways that nobody fully understands.

Immutable infrastructure solves this by refusing to modify servers after they launch. Need a change? Build a new image, launch new instances, terminate the old ones. The fleet renews itself. Every running server traces back to a specific version in the image registry.

The traditional ops model treats servers like pets. You name them, you nurse them back to health when they get sick, you upgrade them in place. The immutable model treats servers like cattle. They have numbers, not names. When one is sick, you replace it.

Netflix pioneered this at scale starting in 2011. Their Chaos Monkey randomly terminates production instances to prove the fleet can absorb the loss. If a service cannot survive random instance death, the service has a design flaw.

The core trade-off: you spend more time building image pipelines up front, but you eliminate an entire class of production incidents caused by configuration drift.