Most reliability engineering is reactive: something fails, and the team responds, as covered in How On-Call and Incident Response Actually Work. Chaos engineering inverts that by asking a specific question in advance, deliberately: if this component failed right now, would the system actually handle it the way we assume it would?
The core idea
Chaos engineering is the discipline of running controlled experiments that deliberately inject failure into a system — terminating a server, introducing network latency, exhausting a resource — specifically to observe whether the system’s actual behavior matches its assumed resilience. Principles of Chaos Engineering, the foundational reference for the discipline (originating from work at Netflix), frames it as an empirical discipline: rather than assuming a system is resilient to a given failure because it was designed to be, chaos engineering tests that assumption directly, in a controlled way, and treats any gap between the assumption and the observed result as a genuine finding to fix.
Why deliberately causing failure is worth doing
The core justification is straightforward: a distributed system’s actual failure modes are often different from, and more numerous than, what was anticipated at design time — the same complexity discussed in What Is Observability? that makes distributed systems hard to debug also makes them hard to fully reason about in advance. Discovering that a particular failure isn’t actually handled gracefully is going to happen eventually, one way or another; chaos engineering’s premise is that it’s far better to discover this deliberately, during a planned, controlled experiment with an engineer watching and ready to stop it, than for the first time during an unplanned, real production outage at 3am.
What makes it “controlled,” not reckless
The discipline of chaos engineering specifically isn’t about randomly breaking production and hoping for useful information. Real practice depends on a few defining constraints:
- A defined, minimized blast radius. Experiments start small — often against a small percentage of traffic or a non-critical component — and expand only once confidence is established, rather than starting with a worst-case, system-wide failure injection.
- A clear hypothesis stated in advance. A chaos experiment tests a specific, falsifiable prediction (“if this service becomes unavailable, traffic should fail over within N seconds with no user-visible errors”), not a vague “let’s see what happens.”
- An abort mechanism. A well-run experiment can be stopped immediately if it starts causing more impact than intended, which depends on having the observability in place to actually notice that in real time.
- Real production, or as close to it as practical. Chaos engineering’s value comes specifically from testing the real system’s actual behavior, including all the accumulated complexity and configuration drift a staging environment often doesn’t fully replicate — which is also why it demands more caution than testing in a fully isolated environment would.
How this connects to error budgets and SLOs
Chaos engineering experiments are a deliberate, controlled way of spending some of the error budget discussed in Understanding SLIs, SLOs, and Error Budgets — intentionally, in service of learning something, rather than involuntarily, during an unplanned incident. Teams with mature chaos engineering practices often treat “have we tested this specific failure mode recently” as a real input into how much they trust a given reliability target, not just a theoretical exercise disconnected from their actual operational numbers.
Where to start
Chaos engineering doesn’t require starting with dramatic, large-scale failure injection. A reasonable starting point is testing a single, specific, already-suspected weak point — a dependency without a documented fallback behavior, a service whose failure mode has never actually been observed — rather than attempting to systematically chaos-test an entire system at once. The goal at every stage is the same: replace an assumption about resilience with an observed, tested result.
Key takeaway
Chaos engineering deliberately injects controlled failure into a system to test whether its actual resilience matches what’s assumed, using a defined blast radius, a stated hypothesis, and the ability to abort — turning an assumption that would otherwise only be tested for the first time during a real outage into something verified in advance, under controlled conditions.
This article explains general chaos engineering concepts; running these experiments safely requires careful planning specific to your system. See our disclaimer.