We cut our alert volume by 70% without missing a real incident
Six months ago, our on-call rotation was miserable. Not because our systems were unstable — they weren't — but because every small blip triggered a page. A queue depth crossing a fixed number for ninety seconds. A single failed health check that recovered on the next poll. By the time something actually mattered, half the team had learned to skim notifications instead of reading them.
That's the real cost of noisy alerting. It's not just wasted sleep, it's the erosion of trust in the system that's supposed to protect you.
What we changed
We stopped alerting on static thresholds crossed once, and started alerting on sustained deviation from a rolling baseline. A queue depth spike that matches Tuesday's normal traffic pattern doesn't page anyone. The same spike at 3 a.m. on a Sunday does.
- Replaced fixed thresholds with baseline-relative rules wherever we could.
- Required two consecutive breaches, not one, before paging for anything non-critical.
- Split alerts into "wake someone up" and "check it in the morning" tiers explicitly.
- Reviewed every page from the previous week, every Friday, and deleted rules that never led to action.
What broke
The baseline approach failed the first time we shipped a genuinely unusual but harmless traffic pattern — a marketing campaign that tripled sign-ups for a day. We paged the on-call engineer for nothing. We fixed it by adding a manual "expected change" window that temporarily widens the baseline, so planned spikes don't get treated as anomalies.
The bigger lesson wasn't technical. It was that alert tuning is never finished. We now treat it as a recurring task, not a one-time project, and revisit the rules every time an incident review turns up a gap.
The result
Page volume dropped by about 70% over two months. More importantly, the pages that remain get taken seriously, because the team has learned they almost always mean something. That trust is the part that's hard to rebuild once it's gone, so we'd rather over-invest in keeping it.