Good monitoring and clear reliability targets, covered in Understanding SLIs, SLOs, and Error Budgets, only matter if something effective happens when they indicate a real problem. On-call and incident response are the practices that turn an alert into an actual fix.
On-call: who responds, and when
On-call is a rotation of engineers who take responsibility, on a scheduled basis, for responding to production alerts outside of normal working hours — typically carrying a pager or phone that receives alerts and being expected to respond within a defined time window. Google’s SRE book discusses the practice directly in its chapter on being on-call, including a point worth taking seriously: on-call load is itself a signal. A rotation that pages constantly, or that regularly wakes people at 3am for issues that don’t need immediate human judgment, usually indicates a reliability or alerting problem that needs fixing, not just an on-call schedule that needs more people added to it — the same repetitive, automatable, no-lasting-value pattern Google’s SRE practice specifically names as toil.
Well-designed alerting pages a human only for things that actually need a human, right now — genuine, user-impacting problems requiring judgment — and handles everything else through automated remediation, or by generating a non-urgent ticket instead of an urgent page. Getting this distinction wrong in either direction has a real cost: paging too aggressively causes alert fatigue and burnout; paging too conservatively means real problems sit unnoticed.
Severity levels: not every incident is the same incident
Mature incident response processes typically define a small number of severity levels — commonly numbered SEV1 through SEV3 or SEV4, though naming varies by organization — that determine how an incident is handled: how quickly it needs a response, who needs to be involved, and whether it needs a dedicated incident commander coordinating the response. A minor, contained issue affecting a small fraction of a non-critical feature doesn’t need the same process as a full outage of a core system, and treating every incident identically either under-reacts to serious ones or creates so much process overhead for minor ones that the process itself gets ignored.
Incident commander: coordination separate from fixing
For significant incidents, many organizations designate an incident commander — a role focused specifically on coordinating the response (communication, tracking who’s doing what, deciding when to escalate further) rather than personally debugging the problem. Separating coordination from hands-on technical work matters because a single person trying to do both, during a serious incident with multiple people investigating in parallel, tends to do neither well: the technical debugging suffers from interruptions, and the coordination suffers from someone too deep in a specific technical thread to keep the bigger picture in view.
Postmortems: learning without blame
After an incident is resolved, a postmortem (also called a retrospective) documents what happened, why, and what will change to prevent a recurrence or reduce its impact next time. Google’s SRE book describes the underlying philosophy in its chapter on postmortem culture: a blameless postmortem process, which focuses on the systemic and process factors that allowed an incident to happen — inadequate testing, a missing alert, an unclear runbook — rather than on which individual made a mistake. The reasoning is practical, not just cultural: engineers who fear personal blame for an incident have a real incentive to hide information or minimize their own role in what happened, which directly degrades the quality of the postmortem and makes the same class of incident more likely to recur. Atlassian’s overview of the postmortem process covers the same core practice from a similar angle.
How this connects to the rest of the deployment pipeline
Incidents don’t happen in a vacuum — they’re often triggered by a recent deployment, which is exactly why techniques like the ones in Blue-Green and Canary Deployments matter: a fast, well-understood rollback path shortens the time between “something is wrong” and “the immediate problem is resolved,” which is usually the first priority during an active incident, well before root-causing exactly why it happened.
Key takeaway
Effective incident response depends on more than just having an on-call rotation: clear severity levels determine how seriously to treat a given incident, an incident commander role keeps coordination and technical debugging from interfering with each other during serious incidents, and a genuinely blameless postmortem process is what turns an incident into an improvement rather than just a bad night and a closed ticket.
This article explains general incident response concepts; specific processes should be adapted to your team’s size, structure, and risk tolerance. See our disclaimer.