Triggers and actions: turning alarms into the right notification

A monitoring system that records every problem and tells nobody is a very detailed diary. Triggers are the part that decides when a problem becomes a message, who gets it, and how loudly. Get them right and the right person hears about the right thing within minutes. Get them wrong and people learn to ignore their phones, which is worse than having no alerts at all.

The sentence model

Almost every alerting system, from a 2000s PHP console to Prometheus, reduces to one sentence:

IF <condition> [for <duration>] THEN <action> [repeat every <interval>] [unless <exception>]

"IF the WAN link at branch 2 is down for 2 minutes THEN page the on-call engineer, repeat every 30 minutes until acknowledged, unless branch 2 is in a maintenance window." Writing each rule as one plain sentence first, before touching any configuration screen, catches most design mistakes. If the sentence sounds silly read aloud, the rule will be silly at 3 a.m.

What the condition looks at

Conditions can be written against three different things, and the choice changes how the rule behaves.

Most of a network's alerting should sit on alarms, because an alarm has a start and an end. The end matters: it lets the system send "resolved", stop repeating, and close a ticket. Event conditions suit audit-style notices. Metric conditions suit trends, where no single poll is wrong but the shape is.

Actions

An action is what happens when the condition holds. The usual menu:

• Email. Universal and slow. Fine for warnings, daily digests and anything a person handles during office hours. Put host, object and state in the subject line, because that is all a phone shows.
• Chat. A message to a team channel through an incoming webhook. Good for awareness, poor for waking anyone.
• Paging or SMS. Through an SMS gateway or a paging service with acknowledgement and escalation built in. Reserve it for problems that need a person now.
• Ticket. Opens or updates a ticket in the helpdesk system. Use a de-duplication key (host plus object plus type), or one flapping port opens fifty tickets.
• Script. Collects diagnostics (a traceroute, interface counters, the last 50 log lines) and attaches them, or attempts a safe fix like restarting a stuck service. Run it as an unprivileged user with a timeout.

Each action should carry enough context to act on without opening the console: what, where, since when, how bad, and a link to the runbook.

Timing: delay, repeat, escalate

Three timers decide whether a rule is useful or noise.

The delay absorbs flaps. A 1 to 3 minute wait before acting on a link-down alarm removes most transient blips without slowing response to a real outage. Nagios expresses this with soft and hard states (max_check_attempts and retry_interval) plus first_notification_delay. Prometheus alerting rules use for:. Zabbix can build it into the trigger expression or the action's first escalation step.

The repeat reminds people of a problem nobody has acknowledged. Every 30 minutes for critical, never for warnings, is a reasonable start. Acknowledgement should stop repeats.

Escalation widens the circle when nobody responds: primary on-call at 0 minutes, secondary at 20, the team lead at 45. It only works if acknowledgement is easy from a phone.

In Prometheus, the delay lives in the alerting rule and grouping, repeats and routing live in Alertmanager:

groups:
  - name: sites
    rules:
      - alert: SiteRouterDown
        expr: probe_success{job="icmp", role="site-router"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Site router {{ $labels.instance }} not answering ping"

route:
  receiver: noc-chat
  group_by: ['alertname', 'site']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="critical"
      receiver: oncall-pager
      repeat_interval: 30m

The receivers themselves are omitted here. See the Prometheus documentation for the full configuration format at the time of writing.

Quiet hours and maintenance

Not every alert deserves the night. A practical split: critical alerts page at any hour, warnings go to chat during business hours and collect into a morning digest otherwise. Most tools support time periods on actions (Nagios notification_period, Alertmanager time intervals on routes).

Maintenance windows are different from quiet hours. A window says "this object is being worked on, do not act on it" and should always have an end time. A window left open by accident is how an outage goes unreported for a weekend. Filters change what operators see; maintenance changes what triggers do. The difference is covered in event filters.

Dependencies: one dead router, one page

When a branch router dies, everything behind it stops answering: two switches, a dozen access points, the printers, the door controller. Without dependencies that is twenty alerts for one fault, and the useful one is buried.

Tell the monitor what sits behind what, and it will mark the children as unreachable instead of down and alert only on the parent. Nagios uses parents on host definitions. Zabbix has trigger dependencies. Alertmanager uses inhibition rules:

inhibit_rules:
  - source_matchers:
      - alertname="SiteRouterDown"
    target_matchers:
      - alertname!="SiteRouterDown"
    equal: ['site']

This silences every other alert carrying the same site label while the site router alert is firing. All of this depends on a dependency map that matches reality, which is why organizing hosts with parents and sites comes before writing clever triggers.

Rule order, AND and OR

Rules that combine conditions need care. "Host group core AND severity critical OR type config change" is ambiguous on paper; check whether your tool binds AND tighter than OR or reads rules top to bottom, and use brackets if it has them. Also check whether evaluation stops at the first matching rule or runs every match. With first-match, put specific rules above general ones. With all-match, make sure two rules do not page the same person twice. Either way, end with a catch-all for critical alarms that no other rule claimed, sent to a chat channel, so nothing falls silently through a gap.

An example rule set for a small network

A head office and two branches, about 40 devices and a handful of servers:

Ten rules cover a network that size. If you have fifty, read alert fatigue: writing alerts people still read before adding the fifty-first.

Testing triggers without paging anyone

1. Copy the rule with a test action that goes to your own email or a test chat channel, never the real pager.

2. Validate configuration offline where your tool allows it:

   promtool check rules sites.yml
   amtool check-config alertmanager.yml
   amtool config routes test --config.file=alertmanager.yml severity=critical site=branch2
   

3. Feed it real input. Send a test trap with snmptrap (see SNMP traps and informs), write a syslog line with logger -p local7.warning -t trigger-test "TEST Gi1/0/24 down", block ICMP to a lab host, or fill a test filesystem with fallocate -l 5G /srv/test/fill.img.

4. Time the whole path, from the fault to the phone buzzing. Include the delay you configured.

5. Clear the fault and confirm the "resolved" message arrives and repeats stop.

6. Remove the test action, then repeat the drill every quarter and after every upgrade of the monitor or the notification path.

A trigger nobody has tested is a guess. The one test that matters is the full chain, end to end, on a quiet afternoon, before the night it is needed.