Alert fatigue: writing alerts people still read six months later

It is 03:12 and the phone buzzes. CRITICAL core-sw-01 CPU 82%. The on-call engineer opens a laptop, looks at a graph, watches the line drop back to 30 percent, and goes back to bed. The same thing happens on Tuesday, and again on Thursday. Some months later the phone buzzes at 03:12 with a message that matters, and it gets the same tired swipe.

That is alert fatigue. The problem is not the number of alerts on its own. The problem is that people have learned, correctly, that most alerts do not mean anything. Once they learn that, no ringtone will save you. The only fix is fewer alerts, each of which deserves the interruption.

I have a strong opinion here, so I will state it plainly: an alert that pages a human must require that human to do something now, and the alert must tell them what. Everything in this piece follows from that sentence.

The one question to ask of every alert

Before an alert goes live, ask: "If this fires at 3 a.m., what will the person do?" There are only three honest answers.

Most noisy setups fail because everything is wired to the first row. A record is not a lesser alert. It is the right home for context, and a filtered console view is a good place to read it (see event filters for building those views without turning them into notifications).

Symptoms first, causes second

A symptom is what users feel: the web app returns errors, VPN logins fail, the branch office cannot reach headquarters, packet loss to the internet is above 2 percent. A cause is why it happens: high CPU, a failed fan, a routing session that flapped, a disk that filled.

Page on symptoms. Investigate causes on dashboards once a symptom has your attention. The reasoning is simple. There are dozens of causes for every symptom, and most of them happen often without hurting anyone. A switch can run its control plane at 85 percent for an hour while forwarding every packet in hardware. If you page on CPU, you page on the 99 times nothing was wrong. If you page on packet loss across the core, you page on the one time something was. The Google SRE material at sre.google makes the same argument for services, and it holds just as well for routers and switches.

There is one honest exception: a cause that predicts a symptom with a deadline. A UPS on battery with 12 minutes of runtime left, or a log disk that will fill before morning, deserves a page even though no user has noticed yet. The test still applies. Someone has to act, and the alert tells them by when.

Severity tiers and routing

Many monitoring tools offer five or more severity levels, and syslog itself has eight (0 to 7). That is fine for classifying events. For notification you only need the three tiers in the table above, and you need a clear mapping from the tool's levels to those tiers.

Routing matters as much as severity. Send each alert to the people who can fix it, not to whoever set up the monitoring server. Switch and circuit alerts go to the network on-call. Database alerts go to whoever owns the database. A shared "alerts@" mailbox that everyone receives is a mailbox nobody reads.

Time of day changes urgency, but it should not change the tier. A failed redundant power supply on a core switch is a ticket at 2 p.m. and a ticket at 2 a.m. If your rule says "page only at night", that rule is really two rules, and one of them is wrong.

Flaps, damping and hysteresis

Two tools stop alerts from firing and clearing every few minutes.

The first is duration. The condition must hold for a while, or for several consecutive polls, before the alert fires. In Prometheus that is the for: clause on a rule. In a polling NMS it is usually "N consecutive failures". Duration costs you detection time, and you should know exactly how much. With a 5-minute poll and three consecutive bad results required, the first bad poll lands 0 to 5 minutes after the problem starts and the third lands 10 minutes after that, so the alert fires 10 to 15 minutes late. If that is too slow, shorten the interval rather than dropping the confirmation. The trade-offs are in choosing a polling interval.

The second is hysteresis: separate thresholds for firing and clearing. Fire at 90 percent disk, clear at 85 percent. Without the gap, a value hovering between 89 and 91 fires and clears on every poll, and each clear is followed by a fresh notification that looks new. Interface flap detection works on the same idea. Nagios, for example, watches the recent history of state changes and stops notifying while the share of changes is above a high threshold, then resumes once it drops below a lower one.

Dependencies: one outage, one page

A branch router dies. Behind it are a switch, two access points, a printer, eight phones and a door controller. Without dependencies you get a page for each of them, and the one page that matters (the router) is buried in the middle.

Every serious tool can model this. Nagios has the parents directive on hosts, so anything behind a dead parent becomes UNREACHABLE rather than DOWN, and you can choose not to notify for that state. Zabbix has trigger dependencies. With Prometheus and Alertmanager you use inhibition rules, which mute alerts while a broader alert with matching labels is firing:

inhibit_rules:
  - source_matchers: [ alertname="SiteUnreachable" ]
    target_matchers: [ alertname!="SiteUnreachable" ]
    equal: [ site ]

Alertmanager also groups alerts that share labels into one notification, which helps when a dependency is missing from the model. The field manual page on triggers and actions covers delays, escalation and dependencies in a tool-neutral way.

Three bad alerts, rewritten

These three are composites of alerts I have seen in many configurations. Each "before" is the kind of rule that trains people to ignore the phone.

1. The CPU alert nobody acts on

Before:

Name:     CRITICAL core-sw-01 CPU
Trigger:  cpu_5min > 80
Tier:     page, 24x7
Message:  core-sw-01 CPU 82%

It fires a few times a week, usually while the nightly config backup runs or while a poller walks a large table. Nobody has ever done anything about it, because there is nothing to do.

After:

Name:     Core switching dropping traffic
Trigger:  packet loss core-sw-01 to distribution switches > 1% for 5 min,
          measured from two separate pollers
Tier:     page
Message:  Users in buildings A and B may see slow or failed connections.
          Control-plane CPU and routing adjacency changes are on the "core" dashboard.
Runbook:  https://wiki.example.com/runbooks/core-packet-loss

The CPU number did not disappear. It became a ticket (above 90 percent for 30 minutes, business hours) and a line on the dashboard, where it helps explain the page instead of being the page.

2. The interface that goes down every evening

Before:

Name:     Interface down
Scope:    every interface on every switch
Trigger:  ifOperStatus != up
Tier:     page
Message:  Interface Gi1/0/17 down on access-sw-3f-02

Every evening around six, people unplug laptops and docks, and access ports drop by the dozen. The rule cannot tell a desk port from an uplink, so it pages for all of them.

After:

Name:     Uplink down: access-sw-3f-02 Te1/1/1 to dist-sw-a (Po10 member)
Scope:    interfaces whose description starts with [UPLINK]
Trigger:  ifOperStatus = down(2) on 2 consecutive 60 s polls
          while ifAdminStatus = up(1)
Tier:     ticket if Po10 still has a member up, page if Po10 is down
Message:  Floor 3 is still connected through the second link.
          No redundancy until this is fixed.

Access ports become records. The uplink rule says what is on the other end, whether users are affected, and why it matters. Checking ifAdminStatus keeps a planned shutdown from paging anyone.

3. The disk that fills and empties every day

Before:

Name:     Disk warning
Trigger:  /var used > 85%
Tier:     page
Message:  /var on syslog-01 at 85%

It fires around 23:00 as logs grow and clears at 03:15 when log rotation runs. After a month nobody looks. Then a switch starts sending debug messages, the disk reaches 100 percent at 02:40, and the alert looks exactly like the other thirty.

After, as a Prometheus rule that pages on the deadline rather than the level:

- alert: SyslogDiskFillingFast
  expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/var"}[6h], 4 * 3600) < 0
  for: 30m
  labels:
    severity: page
  annotations:
    summary: "/var on {{ $labels.instance }} will be full within 4 hours at the current rate"
    runbook_url: "https://wiki.example.com/runbooks/disk-filling"

A second copy of the rule with a three-day horizon goes to the ticket tier. The old static threshold stays as a record, with hysteresis (fire at 95 percent, clear at 90) so it stops flickering.

The review ritual

Alert rules decay. Traffic grows, hardware changes, someone adds a "temporary" rule during an incident and never removes it. A short, regular review is the only thing I have seen keep a rule set honest.

1. Same time every week, 30 minutes, with whoever was on call.
2. Pull every notification from the past week, counted per rule.
3. For each rule that fired, ask: did someone act, and what did they do?
4. Decide one of five outcomes: keep, tune (threshold or duration), demote (page to ticket, ticket to record), delete, or automate the action.
5. Every rule has a named owning team. A rule nobody claims for two reviews in a row is deleted.
6. New rules start at the ticket tier for two weeks. They earn the right to page by proving they fire only when something is wrong.

Track two numbers over time: pages per week, and the share of pages that led to an action. The first should go down and the second should go up. If a single rule wakes someone more than once or twice a week and the answer to step 3 is "nothing", it goes on the agenda before anything else.

The one thing to remember: every page is a promise that waking up was worth it. Keep the promise and people will keep reading.