Events, alarms and severities: reading what the console tells you

Every monitoring system, whatever its vocabulary, keeps two kinds of record. One is a running log of things that happened. The other is a list of things that are wrong right now. Mixing the two up is behind most confusing consoles, so this guide keeps them apart: events first, then alarms, then the severity labels attached to both.

An event is a fact with a timestamp

An event records that something was observed at a particular moment. It does not last; it happened. Events arrive from four places:

• Pollers: a check found a state or value different from last time. An interface went from up to down, disk usage crossed 90 percent, a web check started returning 503.
• SNMP traps: a device reported something on its own (see SNMP traps and informs).
• Logs: a syslog line matched a rule, such as a configuration change or a failed SSH login (see turning syslog lines into events).
• The monitor itself: a poller ran late, the database disk filled up, a notification failed to send.

That last group is easy to forget. If the monitor cannot tell you it is struggling, you find out when the graphs have a hole in them.

The fields an event carries

Products name them differently, but almost every event model has the same core. If you write a script that feeds events in, aim to fill all of these.

Host and object carry most of the weight. An event can only change the state of something if it can be tied to an object the monitor knows. A syslog line saying Interface GigabitEthernet1/0/24, changed state to down does nothing for alarms unless the rule that parsed it produces the same interface name the poller uses. Spend time on naming before you spend it on rules.

Event types

A type groups events that should behave the same way. A sensible starter set:

• State change: something went up, down, degraded or recovered. These open and close alarms.
• Threshold: a measured value crossed a line, such as CPU above 90 percent for 10 minutes. These open an alarm and clear when the value falls back below a lower line, so a value hovering at the limit does not flap.
• Notice: worth recording, with no "fixed" state to return to. A configuration saved, a user logged in.
• Security: authentication failures, port security violations, an unexpected new MAC address.
• Internal: the monitor's own health.

Each type sets defaults: a message template with placeholders like {host}, {object} and {state}, a default severity, whether the event shows in the main console or only in the full log, and whether it may open an alarm. Keeping that logic on the type, rather than on each parsing rule, is what keeps a few hundred rules manageable.

Severity scales

Severity is a label for urgency. The trouble is that several scales are in common use, and two of them run in opposite directions.

Syslog, defined in RFC 5424, has eight levels where a lower number is worse:

Monitoring tools use their own words. Nagios, and the plugins that follow its convention, report OK, WARNING, CRITICAL and UNKNOWN. Zabbix trigger severities run from Not classified and Information through Warning, Average and High to Disaster. The ITU alarm model (X.733), common in carrier and vendor gear, uses critical, major, minor, warning, indeterminate and cleared. Prometheus has no built-in severity; teams add a severity label by convention.

When several sources land in one console you need one internal scale and a mapping into it. Three rules help:

1. Map by meaning, not by number. Cisco-style %LINEPROTO-5-UPDOWN messages are syslog level 5, yet a line protocol change on an uplink matters more than a level 3 error from a chatty process.
2. Keep the scale short. Five levels (critical, major, minor, warning, info) are plenty. Nobody can explain level 6 versus level 7 of a twelve-step scale in the middle of the night.
3. Set severity by type and by object role. A link down on a desk phone's access port is minor; the same event on a core uplink is critical. Most tools can raise severity by host group or interface tag.

How events open and close alarms

An alarm is a standing condition on one object: this interface is down, this disk is above 90 percent. It has a start time, a current severity and, eventually, an end time. Events open it, update it and close it.

A worked example: a distribution switch uplink that flaps once, then fails properly.

Three mechanisms are at work there:

• Matching. Each event is tied to the alarm by host, object and type. The poll result and the syslog line must resolve to the same interface name as the trap, or you get three separate alarms for one cable.
• De-duplication. Repeat events for an open alarm update it (a count, a last-seen time) instead of adding rows.
• Clearing. A matching event with a good state closes the alarm. For types with no natural "good" event, such as authentication failures, set an expiry so the alarm closes itself after, say, 30 minutes of quiet.

The 12-second flap at 02:41 belongs in the record but should not wake anyone. Absorbing it is the job of a delay in your triggers, not of the event log.

Acknowledgement

Acknowledging an alarm means "a person has seen this and owns it". It should record who, when and a short comment ("ticket 4471, sending someone to the site"). It should not clear the alarm, because the problem is still there. In most tools an acknowledgement also stops repeat notifications, which is the main reason operators use it.

Two habits worth copying from good systems: drop the acknowledgement if severity rises, since a minor problem that became critical deserves fresh eyes, and show acknowledged alarms dimmed rather than hidden. Hiding is a job for filters, chosen on purpose.

Retention

Events pile up. A few hundred devices with chatty syslog can produce hundreds of thousands a day. A plan that suits most small and mid-sized networks:

• Raw events in the live database: 30 to 90 days, purged nightly at a quiet hour.
• Alarm history (open time, close time, object, severity): at least 13 months, so you can compare a month with the same month last year and build availability reports.
• Anything you must keep longer goes to a log store or compressed archive files, not the live tables.

Index on time, and on host plus time. Most slow consoles are slow because a query scans months of events to show the last hour. Purge jobs and database care are covered in running the system day to day.

A severity map for common events

A starting point; adjust by host role.

If you remember one thing from this page: an event is a line in the diary, an alarm is an item on the to-do list. Keep the diary long, keep the to-do list short, and make sure every event that should touch the list can find the right item on it.