A naive retry policy — retrying a failed request immediately, without limit — can turn a struggling but recoverable service into a fully failed one, by piling additional retried load onto exactly the service that's already overwhelmed; circuit breakers stop sending requests to a failing dependency for a cooldown period, and exponential backoff with jitter spreads out retries over time, both specifically to prevent this feedback loop.
reliabilitydistributed-systems
A runbook is a documented procedure for diagnosing or resolving a specific, known operational scenario; the difference between a runbook that actually gets used during an incident and one that gets ignored usually comes down to whether it's kept current, tested against reality, and written for someone acting under real time pressure rather than for completeness.
reliabilityon-call
Alert fatigue happens when the volume of low-value or non-actionable alerts is high enough that engineers start treating all alerts, including genuine ones, as background noise; the fix isn't more alerts or more sensitive thresholds, it's fewer, better ones — alerting only on symptoms that need a human right now, tied to concrete signals like the four golden signals or a defined error budget.
reliabilityon-call
The four golden signals — latency, traffic, errors, and saturation — are a practical starting framework for deciding what to monitor on any user-facing service; together they cover how fast requests are handled, how much load the system is under, how often things fail, and how close the system is to its capacity limits, which is a more disciplined starting point than monitoring whatever metrics happen to be easiest to collect.
observabilitymonitoring
A distributed trace follows a single request as it moves through multiple services, made up of individual timed spans linked together by shared trace context that's passed along with the request; OpenTelemetry is the vendor-neutral standard that defines how this data is generated and exported, which matters because it prevents instrumentation from locking an application into one specific observability vendor.
observabilitytracing
Chaos engineering is the discipline of deliberately, carefully injecting failure into a system in a controlled way to test whether it actually behaves as resiliently as assumed; the point isn't to cause damage, but to find weaknesses in a planned experiment, with a defined blast radius and an abort mechanism, rather than discovering them for the first time during a real, unplanned outage.
reliabilitychaos-engineering
On-call is a rotation of engineers responsible for responding to production alerts outside normal hours; effective incident response also depends on clear severity levels, a defined incident commander role during major incidents, and a blameless postmortem process that treats failures as system problems to learn from rather than individual mistakes to punish.
reliabilitysre
An SLI is a measured signal of service health (like request latency or error rate), an SLO is a target value for that signal over time, and an error budget is the amount of allowed unreliability implied by the gap between an SLO and perfect performance — a budget that, once spent, is meant to shift priority toward reliability work.
reliabilitysre
Observability is the property of being able to understand a system's internal state from its external outputs, including questions you didn't anticipate in advance; logs, metrics, and traces are the three complementary data types that make that possible, and each answers a different kind of question.
fundamentalsobservability
