How monitoring data flows from raw input to alarms and actions

Most confusion about configuring a monitoring system comes from not knowing where a piece of data is in its life. Is that red line on the console a raw syslog message, an event, or an alarm? Why did one cable pull produce four entries in one view and a single entry in another? Why did nobody get paged?

Every NMS moves data through the same five stages, whatever names its menus use:

poll results  \
SNMP traps     >  raw input -> normalized events -> consolidation -> alarms -> actions
syslog lines  /

This chapter follows one ordinary failure through all five, then lists the things you configure and the order that saves you rework.

The running example

A wireless access point in a school library hangs off port Gi1/0/17 of sw-lib-acc2, an access switch at 10.20.6.21. The access point draws power over Ethernet from that port. At 09:14 on a Tuesday the cable, which runs under a rolling book cart, gets crushed. The port goes down.

The monitor sits at 10.20.0.50. It polls interface status every 60 seconds and receives traps and syslog from every switch.

Stage 1: raw input

By 09:15 the monitor holds four separate pieces of raw input about one fact, three of them from the first two seconds. The first is the linkDown trap, as snmptrapd logs it:

2026-03-10 09:14:02 sw-lib-acc2 [UDP: [10.20.6.21]:50812->[10.20.0.50]:162]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (183022411) 21 days, 4:23:44.11
SNMPv2-MIB::snmpTrapOID.0 = OID: IF-MIB::linkDown
IF-MIB::ifIndex.10017 = INTEGER: 10017
IF-MIB::ifAdminStatus.10017 = INTEGER: up(1)
IF-MIB::ifOperStatus.10017 = INTEGER: down(2)

Then two syslog lines, one for the physical link and one for the line protocol:

Mar 10 09:14:02 10.20.6.21 214: Mar 10 09:14:01.884: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/17, changed state to down
Mar 10 09:14:03 10.20.6.21 215: Mar 10 09:14:02.886: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/17, changed state to down

And at 09:15:00, the scheduled poll:

IF-MIB::ifOperStatus.10017 = INTEGER: down(2)

Raw input is unprocessed and unmatched. It belongs to no object yet. Notice that the trap and the poll identify the port by its SNMP index (10017), while syslog names it GigabitEthernet1/0/17. Matching those to one object is the next stage's job. The linkDown notification and the ifIndex, ifAdminStatus and ifOperStatus values it carries are defined in the IF-MIB, RFC 2863.

Stage 2: normalized events

Normalization turns each raw message into an event with the same small set of fields, no matter where it came from.

Three things make this stage work. The monitor needs a host lookup that maps the sender's address to a known host. It needs an object lookup per host, which for interfaces means a current table of index, ifName and ifDescr (refreshed by discovery, because indexes can change after a reboot or a module swap). And it needs rules that recognize each raw message: trap OIDs mapped to event types, and regular expressions for log lines, covered in the chapter on turning syslog lines into events.

Anything that fails to match still becomes an event, flagged as unmatched or unknown. Never drop it silently. An unexplained burst of unknown events is often the first sign that a firmware upgrade changed a message format.

Stage 3: consolidation and de-duplication

Four events now describe one fact. Consolidation decides which events are the same story, and de-duplication folds them together.

The usual key is host plus object plus type plus state. Events that share a key within a short window collapse into one, with a counter and a list of sources. The %LINEPROTO-5-UPDOWN line is a slightly different message about the same failure, so a sensible rule set treats it as supporting evidence rather than a second event type. The result is one consolidated record: Gi1/0/17 on sw-lib-acc2, link down since 09:14:01, seen 4 times, by trap, syslog and poll.

Why bother? Without consolidation, every source multiplies the noise. One cable pull would mean four console rows, four notifications and four entries in the availability report. Multiply by a core switch reboot that takes 300 ports with it, and the console becomes useless at exactly the moment you need it.

Consolidation is also where dependencies apply. If the distribution switch above sw-lib-acc2 were down, the access switch's own alarms would be marked as consequences of that parent failure instead of raised on their own. How to model parents and children is part of organizing hosts.

Stage 4: alarms and the state table

An alarm is the consolidated problem attached to an object, with a start time and, eventually, an end time. Behind it sits a state table: one row per object, showing its current state and, when it is bad, the alarm that explains it.

object                    type  state  alarm  opened    closed    ack   count
sw-lib-acc2 Gi1/0/16      link  up     -      -         -         -     -
sw-lib-acc2 Gi1/0/17      link  down   A-8812 09:14:01  -         no    4
sw-lib-acc2 Gi1/0/18      link  up     -      -         -         -     -

The rules are short. An event with a bad state opens an alarm if none is open for that object and type; otherwise it increments the open one. An event with a good state (here a linkUp trap, a "changed state to up" log line, or a poll returning up(1)) closes the open alarm and stamps the end time. The difference between those two timestamps is what availability reports add up later. The host's color on the map is the worst state among its objects, and the map's color is the worst among its hosts.

Severity and event type definitions live in the chapter on events, alarms and severities.

Flapping

Now the twist. The crushed cable is not cleanly broken; it makes contact when the cart moves. The access point keeps rebooting, and the port goes down and up 14 times in 20 minutes.

Without flap handling, that is 14 alarms opened and closed, and if notifications fire on both, 28 messages to whoever is on call. With flap handling, the monitor counts state changes per object over a window. Past a threshold, it stops treating each change as news, marks the object as flapping, raises a single "flapping" alarm, and holds individual notifications until the object has been stable for a set time. Nagios and Icinga, for instance, compute a percent state change across recent checks and compare it with a high and a low threshold, so the object has to settle down properly before it leaves the flapping state.

A simpler tool that lacks flap detection can get most of the benefit from a delay on notifications: only act if the alarm is still open after 2 or 3 minutes. A flapping port then produces at most one message per long outage instead of one per bounce. The console still shows the history, which is exactly what you want an engineer to read.

Stage 5: actions

Actions run against alarms far more often than against raw events, because an alarm has a lifetime to reason about: how long it has been open, whether someone acknowledged it, whether it closed by itself. In our example a sensible rule says that access ports raise a console alarm only, uplinks page after 2 minutes, and any object flapping for more than 30 minutes opens a ticket for the morning. Writing those rules, with escalation, quiet hours and dependencies, is the subject of the chapter on triggers and actions.

What to configure, and in what order

Each item below depends on the ones above it. Configure in this order and you avoid redoing work.

1. Credentials and protocol profiles. SNMPv3 users or restricted v2c communities, ping settings, timeouts. Discovery cannot work without them.
2. Sites and host groups. Buildings, regions, customers, roles. Permissions, maps and reports all hang off these.
3. Hosts. Names, addresses, group membership, parent relationships.
4. Objects. Run discovery to find interfaces, disks and sensors, then switch off what you do not want watched (unused ports, loopbacks, virtual interfaces).
5. Templates or poller groups. Which values to collect for each object type, how often, and with which thresholds.
6. Event types and severities. The vocabulary the next step maps into.
7. Input rules. Trap OIDs and syslog patterns mapped to event types, with captures for host, object and state.
8. Consolidation settings. De-duplication windows, flap thresholds, dependencies.
9. People and contacts. Users, teams, on-call rotations, how each person is reached.
10. Triggers and actions. Only now, when events are clean and alarms are trustworthy.
11. Maintenance windows. So planned work does not page anyone or count against availability.

Most teams do step 10 first because notifications feel like the point. Resist it. A trigger built on unmatched events and duplicate alarms will page people for noise, and once people learn to ignore the pager, getting their trust back takes far longer than doing steps 1 to 9 properly.