How do we set up monitoring that people actually pay attention to?

Every organisation with alert fatigue built it the same way: an alert per metric, added one incident at a time, never removed. The fix is a different principle, not more tuning.

Alert on what the user feels

Page a human when requests are failing, when they are slow enough to matter, or when a queue is growing faster than it drains. Do not page on high CPU — high CPU is only a problem if it is causing one of the above, and if it is, the symptom alert already fired.

Separate pages from tickets

Two tiers only. A page means someone wakes up right now. Everything else creates a ticket for working hours. Most alerts currently paging in a typical setup belong in the second tier.

Every alert needs a runbook

If an alert fires and the responder does not know what to do, the alert has failed regardless of whether it was correct. Link each alert to a short document: what it means, what to check first, how to mitigate, who to escalate to.

Instrument before you need it

Structured logs with a request identifier that follows a call across services, plus traces on the paths that matter. Adding this during an incident is not possible, which is when everyone wishes they had it.

Review alerts monthly

Pull the list of every alert that fired. Any that resolved itself without action gets deleted or downgraded. This one habit does more for signal quality than any tooling change.