everything fails, all the time

cloudops culture

Werner Vogels' cloud axiom: at sufficient scale, failure is not an event but a permanent background condition.

With enough disks, links, and instances, something is always broken - so architecture must assume it: redundancy, graceful degradation, retries with backoff. The mindset shift is from preventing failure to making it boring.

Werner Vogels's formulation is a design premise rather than a complaint. Every component will fail, at some rate, and at sufficient scale that rate becomes a continuous condition rather than an occasional event: with enough machines, something is always broken right now.

That reframing changes what a system is for. If failure is exceptional, the sensible response is to prevent it and handle the exception when prevention fails. If failure is constant, prevention is not the primary strategy, and the system has to keep working while components are failing, which means health checking, automatic replacement, retries, timeouts and graceful degradation are core functionality rather than error handling.

It also changes what quality means. A design is not judged by whether it fails but by how it fails: whether a failure is contained or propagates, whether it is visible, whether recovery is automatic, and whether the failure of a dependency degrades service or removes it. Chaos engineering follows directly, since the only way to know how a system behaves under failure is to fail it deliberately while people are watching rather than waiting to find out.

← All glossary entries