Alerts That Earn the 2am Wake-Up
Alert fatigue is what happens when your monitoring cries wolf so often that the team stops listening, and then the real outage pages into silence. Every on-call engineer knows the pattern: hundreds of notifications a week, most of them informational, a few of them stale, and one buried page that actually mattered. This guide fixes that. You will learn which four things deserve to wake a human, how to write an alert that carries its own fix, and how to run an on-call rotation your team does not dread.
The principle is blunt. An alert that does not require immediate human action is not an alert. It is a log message wearing a costume. Every page should mean: something users feel is broken, and a person must do something now. Everything else belongs in a dashboard, a daily digest, or nowhere.
Here is where we are going:
How alert fatigue creeps in
The only four things worth paging for
Write alerts that carry their own fix
The monthly ritual that kills noise
On-call rotations humans survive
Measuring whether alerting works
Connecting alerts to response
How Alert Fatigue Creeps In
It starts innocently. Someone adds an alert for disk usage at 80 percent, just in case. Someone else pages on every deploy. A third person copies a monitoring template with forty checks enabled. Each one seemed reasonable alone. Together they produce a phone that buzzes all night for nothing, and a team that learns the rational response: ignore it.
That learned ignoring is the actual danger, not the noise itself. Studies of alarm-heavy environments keep finding the same result: when most alarms are false, operators miss the real ones. Your team is not lazy for muting the channel. They adapted to a system that trained them to. The fix is never discipline or louder ringtones. The fix is fewer, better alerts, each one earning its interruption.
Count your pages for the last month before changing anything. Total pages, pages that required action, pages that were false or stale. Most teams discover an action rate under a quarter, and some under a tenth. That number is your baseline, and moving it is the whole project. If every alert demanded action, nobody would talk about fatigue, because there would be nothing to be fatigued by.
The Only Four Things Worth Paging For
Narrow the paging bar to four categories and push everything else down to tickets, digests, or dashboards.
User-facing pain, now. Errors, latency, or downtime customers feel this minute. Checkout failing, logins broken, pages not loading. If users would call support about it, it pages.
Data loss risk. Backups failing, replication lag growing past the point of no return, disks filling on the database primary. Silent data problems become loud ones eventually, so these page early.
Active security breach. Intrusion detected, credentials leaked, suspicious privilege changes. Speed matters more than certainty here, so these page on suspicion, not proof.
Cascading failure in progress. One service dying is an incident. Three dying together is a cascade, and cascades do not fix themselves. Page on the pattern, not just the parts.
Notice what is missing: CPU at 85 percent, a single failed background job, staging errors, certificate expiring in thirty days. Those deserve tracking, tickets, or a morning digest, not a 2am page. A useful test for any alert: if this fires at 3am, will the woken engineer do something different than in the morning? If not, it is not a page.
Write Alerts That Carry Their Own Fix
Every alert should arrive with three things attached: what is wrong, why it matters, and where the fix lives. An alert that says only "ERROR RATE HIGH" sends a half-asleep human hunting through dashboards. An alert that links its runbook turns the same wake-up into a procedure.
Here is the shape, using a generic monitoring rule as the example:
- alert: CheckoutErrorsHigh
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
for: 5m
labels:
severity: page
annotations:
summary: "Checkout 5xx rate above 5 percent"
runbook: "https://wiki.example.com/runbooks/checkout-errors"Read it as a contract. The expression defines the symptom precisely, the for clause waits five minutes so brief blips do not page, the severity says this one wakes people, and the annotations hand the responder the summary and the exact runbook page. That runbook, written the way the support runbooks guide describes, holds the symptoms, the ordered checks, and the escalation path. Alert plus runbook is the whole response system in two linked pages.
Severity levels need shared meaning or they are decoration. Use three and enforce them: page (act now, wake up), ticket (act today, business hours), log (act never urgently, review in digests). Every new alert must pick one, and anything that cannot justify "page" drops a level. Revisit the levels quarterly, because urgency drifts upward whenever nobody pushes back.
The Monthly Ritual That Kills Noise
Alerts rot. Thresholds set for last year's traffic misfire on this year's growth, deploys change behavior, and temporary alerts outlive the incidents they were created for. Without maintenance, any alert set degrades back into noise within a year. The counterweight is a short monthly review with teeth.
Pull three lists: alerts that fired most, alerts that fired with no action taken, and alerts nobody can explain anymore. For each top offender, choose one outcome: tune it (raise the threshold, lengthen the wait, narrow the scope), demote it (page becomes ticket, ticket becomes digest), or delete it. Deletion is a valid outcome and should be the most common one. Most noise comes from alerts whose moment passed, and the kindest thing is a quick death.
Track who owns each alert while you are there. Ownerless alerts are where staleness hides, because nobody feels responsible for the noise. The owner does not babysit the alert daily. They just confirm quarterly that it still fires only when it should. The same ownership habit keeps runbooks alive, and for the same reason: visible ownership gets maintained, anonymous pages rot.
Celebrate the kills. Report monthly how many alerts died and how the action rate moved. Teams that watch noise shrink defend the ritual. Teams that never see the trend let it slide, and the slide is how you got here.
On-Call Rotations Humans Survive
Even perfect alerts hurt if the rotation is cruel. Design the schedule for sustainable humans, not idealized ones.
Keep rotations short enough to stay sharp and long enough to learn the system: one week is the usual sweet spot. Follow every rotation with real recovery time, and never stack on-call with deadline weeks or holidays if you can avoid it. Pay or compensate the burden explicitly, in money or time off, because invisible labor breeds quiet resentment that surfaces as turnover.
Hand off like a relay, not a cliff. The outgoing engineer briefs the incoming one: what is flaky right now, which alerts are new, what changed this week. Fifteen minutes of overlap prevents the classic failure where Friday's deploy pages Saturday's engineer with zero context. Write the handoff down in the channel so the night shift can find it later.
Protect sleep as a reliability investment, not a perk. A responder woken three times discards judgment by morning, and bad judgment during incidents costs more than any schedule convenience saves. If one service pages weekly, the problem is the service or its alerts, not the rotation. Fix the source. And back everyone up: no single point of on-call failure. Every rotation needs a secondary who can step in, and every secondary needs enough context to actually help.
Measuring Whether Alerting Works
Three numbers tell you if the system is healthy. Page volume per week shows the load on humans; it should trend down after the cleanup and stay flat as you grow. Action rate, the share of pages that required action, is the quality score; aim for the large majority, and investigate any slide. Time to acknowledge and time to mitigate show whether pages still produce fast, effective response; if acknowledgment slows, the team may be tuning out again.
Borrow the CI mindset: alert changes go through review like code changes. A new page-worthy alert needs a second pair of eyes, a linked runbook, and an owner before it ships. Review the numbers from the support metrics guide alongside these, because alerting and support feel the same incidents from opposite ends. When both improve together, the system is genuinely getting quieter and faster, not just shifting pain around.
Connecting Alerts to Response
Zoom out and the full incident lifecycle is one connected system. Monitoring detects, alerts route, the outage playbook responds, and the postmortem feeds both the runbook and the alert set. Each link strengthens the next: good alerts trigger practiced responses, practiced responses produce accurate postmortems, and postmortems tune the alerts that started it all.
Start this week with the count. Pull last month's pages, compute the action rate, and kill the three noisiest offenders. One month of the ritual, and the 2am wake-ups that remain will be the ones worth waking for. That is the whole promise of this work: fewer interruptions, each one earned, answered by people who trust the sound.
