Every team that has run a system for more than a year has a channel full of alerts nobody reads. The usual response is a renewed commitment to pay attention. That never works, for a structural reason: human vigilance for low-signal, high-volume streams degrades predictably. If your alerting depends on people fighting that mechanism, it will fail.
How teams get here
Nobody designs a bad alerting setup; it accretes. An incident happens, someone says "we should have known sooner," an alert is added, and nobody ever deletes anything. The second driver is alerting on causes rather than symptoms — CPU above 80%, memory above 90% — which fire constantly and train the team that red means "probably fine." The third is duplication.
The test for whether an alert should exist
If this fires at 3am, is there something a human must do right now? If no, it should not page anyone — route it somewhere quieter. Any alert that has fired more than a handful of times without anyone acting is noise wearing an alert's clothing.
Symptoms over causes
The highest-leverage change most teams can make. Alert on checkout error rate, API latency past the point users abandon, a job that did not run. Do not page on CPU, memory, disk I/O or pod restarts. Where a resource genuinely predicts failure, alert on the trajectory — "disk will be full in four hours" is actionable; "disk is at 76%" is not.
Deduplication does real work
A single outage should produce one alert, not eleven, and a six-hour outage should produce one alert plus one recovery, not three hundred and sixty.
The audit that fixes it
Export every alert that fired in the last thirty days, grouped by rule with counts. For each, how many led to someone doing something? The top of that list is dominated by rules with a near-zero action rate — deleting the top five typically cuts volume by more than half. Adopt the rule: for every alert you add, delete one.
What good looks like
A healthy paging setup wakes someone a few times a month, and nearly every wake-up involves real work. That assumption — that a page means something real — is the entire asset, takes months to build, and can be destroyed in a week by one noisy rule left in place.