The First Five Minutes of an Outage: What to Do Before You Know What's Wrong
A production outage pages you, your heart rate doubles, and every instinct screams to start debugging immediately. Resist that instinct. The first five minutes of an outage are not for diagnosis. They are for confirming the problem is real, getting the right people in one place, stopping the bleeding, and telling everyone what is happening. This guide gives you that sequence, step by step, so the worst minutes of your on-call rotation run on procedure instead of adrenaline.
The principle behind everything here: stabilize first, diagnose second. A broken site that is stable, with customers informed and the team assembled, is a problem you can solve. A broken site with three people debugging three theories in silence while support tickets pile up is chaos with a root cause. The five minutes below buy you the calm that makes the next fifty productive.
Here is the sequence:
Verify it is real
Declare the incident and assign roles
Stop the bleeding
Communicate while you are still blind
Hand off to real diagnosis
Close out with a postmortem and a runbook update
Practice before you need it
Minute Zero: Confirm the Production Outage
Alerts lie just often enough to matter. A failed health check can mean the monitoring probe lost its network, not that production is down. A flood of error notifications can be one user hammering refresh. Spend the first sixty seconds confirming reality before you wake anyone up.
Check from the outside first. This one command tells you what a visitor actually gets:
curl -s -o /dev/null -w "%{http_code}\n" https://your-site.com/healthIt fetches your health endpoint silently and prints just the status code. A 200 with a healthy body means the site answers. Anything else, or a hang, confirms trouble. Replace the URL with your real health check, ideally one that exercises the database and not just the web server, since a site that serves 200s with a dead database is down in every way that counts.
Then check scope: is it everyone or someone? One region, one endpoint, one customer tier, or everything. Scope decides who needs waking and how loud the response should be. A single broken report page is a ticket. Everything returning errors is an incident. Write down what you observe with timestamps. These notes become the first lines of the incident record, and future you will be grateful.
Declare It and Assign Three Roles
If it is real, say so out loud in your team channel with one message: what is broken, since when, and that you are declaring an incident. "Checkout returns 500s since 02:14, declaring incident, opening channel #incident-241" takes ten seconds and prevents the worst failure mode of outages: five skilled people working alone in five direct messages.
Then assign three roles, out loud, by name. The commander owns the response: they decide, prioritize, and keep time, and they do not touch keyboards. The fixer investigates and acts. Comms handles every update, internal and external. One person can hold commander plus one other hat on a small team, but never let the fixer also be comms. The person typing commands cannot also write calm status updates, and status updates are how the rest of the company stays out of your way.
If you are alone, you are all three, in rotation. Spend two minutes stabilizing, one minute writing an update, repeat. The rotation feels slow, but it prevents the classic solo failure: forty silent minutes of debugging followed by an angry leadership channel asking what is happening.
Stop the Bleeding Before You Diagnose
Here is the discipline that separates experienced responders from everyone else: restore service first, understand later. Diagnosis feels productive, but every minute of clever investigation is another minute of downtime. Ask one question: what is the fastest path back to working, even partially?
The usual answers, in order of preference. Roll back the last deploy if the timeline fits; most outages follow changes, and undoing the change is the highest-probability fix in existence. Scale out if traffic or load is the story; extra capacity buys room to think. Shed load or degrade gracefully if you must: disable the expensive feature, serve cached pages, pause the background jobs hammering the database. A site running at reduced function beats a site running not at all, and customers forgive "some features are temporarily limited" far faster than another hour of errors.
Resist the urge to try the clever fix first. Clever fixes are for staging. The outage window is for the boring, reversible move with the highest chance of restoring service. Write down what you tried and when, because this log becomes the postmortem timeline for free.
Communicate While You Are Still Blind
The hardest updates are the ones with nothing to report yet, and they are the most important. Silence during an outage is read as either cluelessness or indifference, internally and externally. Update on a timer, not on news: every 15 minutes, no matter what.
Each update has three lines. What we know: "checkout returns 500s since 02:14, affecting all regions." What we are doing: "rolled back the 02:00 deploy, watching error rates." When the next update lands: "next update at 02:45." No speculation, no blame, no promises about fixed-by times you cannot keep. "We do not know the cause yet" is a respectable status when paired with visible action and a next check-in.
Put the update where people already look. Status page for customers, incident channel for the company, and one person answering support so agents have something true to say. Your support team will get the same question fifty times while you work. One accurate sentence from comms, refreshed every fifteen minutes, turns fifty panicked tickets into fifty calm ones. The difficult-customer guide covers the tone; this is the pipeline that delivers it.
Hand Off to Real Diagnosis
Once service is restored or stabilized, and only then, the fixer gets to be curious. The handoff from response to diagnosis needs the same discipline as everything before it: state what is known, what was tried, and what the current theory is, in writing, in the channel.
Known: error rates, scope, timeline, which mitigation is holding.
Tried: rollback, scaling, feature flags, with timestamps and results.
Theory: the current best guess and the single next test that would confirm or kill it.
One theory at a time. Parallel theories across three people produce contradictory changes and destroy the timeline. The commander picks the most likely theory, the fixer tests it, comms keeps updating. If the theory dies, pick the next one. This loop feels slower than swarming, but it converges, and convergence is what ends incidents.
Keep the mitigation in place until the diagnosis is confirmed and the real fix is verified. Rolling forward past a band-aid because things look quiet is how outages recur at 4am. Boring wins again.
Close Out With a Postmortem and a Runbook Update
Every incident buys knowledge at full retail price. Collect it while it is fresh with a blameless postmortem: timeline from your notes, what detected it, what mitigated it, what the root cause was, and what changes so it cannot recur or gets caught faster. Ask what allowed it, never who caused it. People who feel safe describe accurately, and accuracy is what prevents repeats.
Then convert the learning into the two artifacts that compound. First, update or create the runbook for this failure: the symptoms to recognize it, the mitigation that worked, the escalation path. Second, file the prevention work as real tickets with owners, not as vague intentions. "Add alerting on checkout error rate" with a name and a date gets done. "We should monitor better" evaporates by Friday.
Share the postmortem with the wider team, including support. Support agents who understand what broke answer the next similar ticket with confidence instead of fear, and the incident record becomes training material that outlasts everyone in the channel that night.
Practice Before You Need It
Nobody learns incident response during their first real outage. Teams that look calm have rehearsed, usually with game days: scheduled exercises where someone injects a failure into staging and the team runs the sequence. Kill the database primary, blackhole a region, expire a certificate. The point is never the specific failure. It is practicing the choreography: declare, roles, stabilize, communicate, diagnose, close out.
Start small. A tabletop walkthrough takes an hour: pick last quarter's worst incident and talk through what the first five minutes should have looked like. Then run a live drill quarterly. Each rehearsal surfaces a gap cheaply: the runbook with the dead link, the alert nobody receives, the new hire who does not know the channel name. Fixing those on a calm afternoon is the entire return on investment.
Five minutes, practiced. That is the whole skill. Verify, declare, stabilize, communicate, then diagnose with a team that trusts the process. The outage will still be stressful. But it will also be short, orderly, and over, with a better runbook waiting on the other side.
