Feeding your monitor from outside programs

Polling is the monitor asking questions on its own schedule. A lot of the useful information on a network does not wait to be asked: a switch sends a trap the moment a link drops, a router logs a configuration change, a backup job finishes at 03:15, a cloud service fires a webhook. Getting those into the monitor means running, or listening to, programs outside it.

There are four doors, and almost every integration uses one of them:

1. A daemon receives something (a trap, a syslog line) and hands it to a script or forwards it.
2. The monitor runs a program and reads its exit code and output (an active external check).
3. A program elsewhere pushes a result into the monitor (a passive check).
4. Another system calls the monitor over HTTP (an inbound webhook).

Whatever arrives through any of them should end up as a normal event with a host, an object, a state and a message, as described in events, alarms and severities. If an input cannot be mapped to those fields, it is not ready to be wired in yet.

Traps through snmptrapd

The net-snmp trap daemon listens on UDP 162 and can run a program for each trap it accepts. Two lines in snmptrapd.conf are enough to start:

authCommunity log,execute n0c-tr4p
traphandle default /usr/local/libexec/nms/trap-in.py

authCommunity says which community may send traps and what snmptrapd may do with them; without execute, the handler never runs. For SNMPv3 traps you create a user and use authUser instead, and the user must be created with the sending device's engine ID, which is why many people prefer informs for v3. The SNMP traps chapter covers trap types and testing.

The handler receives the trap on standard input: the sender's host name on the first line, its transport address on the second, then one variable binding per line. A minimal handler that drops each trap into a spool directory for the monitor to collect:

#!/usr/bin/env python3
import json, sys, time, uuid

lines = sys.stdin.read().splitlines()
event = {
    "received": int(time.time()),
    "host": lines[0] if lines else "",
    "transport": lines[1] if len(lines) > 1 else "",
    "varbinds": [line.split(" ", 1) for line in lines[2:]],
}
with open(f"/var/spool/nms-in/trap-{uuid.uuid4().hex}.json", "w") as f:
    json.dump(event, f)

Writing to a spool and returning at once matters. A handler that talks to a slow database can hold up trap processing, and traps that arrive during a storm are the ones you most want to keep.

Syslog through rsyslog or syslog-ng

Network devices send syslog over UDP 514 (or TCP where the device supports it). On Cisco IOS, for example, logging host 10.10.0.50 and logging trap informational send everything at informational and above to the monitoring server. From there you have two options: forward the stream to a log store or to the monitor's own listener, or hand selected lines to a program that turns them into events.

With rsyslog, the omprog module pipes matching messages to a long-running program, one message per line:

module(load="imudp")
input(type="imudp" port="514")
module(load="omprog")

if $syslogfacility-text == "local6" then {
    action(type="omprog" binary="/usr/local/libexec/nms/syslog-in.py")
    stop
}

The same idea in syslog-ng uses a program() destination with a template, so the script receives exactly the fields it expects:

source s_net { network(transport("udp") port(514)); };
filter f_nms { facility(local6); };
destination d_nms { program("/usr/local/libexec/nms/syslog-in.py" template("$ISODATE $HOST $MSG\n")); };
log { source(s_net); filter(f_nms); destination(d_nms); };

In both cases the program reads standard input in a loop and must keep running; if it exits, the daemon restarts it, and anything sent in between may be lost. Turning raw lines into structured events with regular expressions has its own chapter, syslog lines to events, and syslog for small networks covers collection and rotation.

TACACS+ accounting logs

If your network devices authenticate administrators against a TACACS+ server (the protocol is documented in RFC 8907), the same server can log every command they type. On Cisco IOS:

aaa accounting exec default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+

Each accounting record carries a time, the device address, the user name, the terminal line, the client address, a start or stop marker and attribute pairs such as service=shell and cmd=.... The exact file layout depends on the TACACS+ server you run, so read a few lines of your own log before writing a parser. A small script that follows the file and raises an informational event whenever a cmd= value starts with configure or write gives the event console a "who changed what, where" trail. It complements a configuration backup tool that diffs the running config; it does not replace one.

External checks and the exit-code convention

The most widely shared convention comes from Nagios plugins and is also used by Icinga, Naemon, Checkmk and Sensu. The monitor runs a program, reads the first line of output as the status message, and takes the state from the exit code:

Text after a | on that first line is performance data, in the form label=value[unit];warn;crit;min;max, which most tools graph automatically. Here is a complete check for the age of the newest file in a backup directory (it uses GNU find, so it is for Linux):

#!/bin/bash
dir=${1:-/srv/backups}
warn_h=${2:-26}
crit_h=${3:-50}

newest=$(find "$dir" -type f -printf '%T@\n' 2>/dev/null | sort -n | tail -1)
if [ -z "$newest" ]; then
    echo "UNKNOWN: no files found in $dir"
    exit 3
fi

age=$(( $(date +%s) - ${newest%.*} ))
hours=$(( age / 3600 ))
perf="age=${age}s;$(( warn_h * 3600 ));$(( crit_h * 3600 ));0"

if [ "$hours" -ge "$crit_h" ]; then echo "CRITICAL: newest backup is ${hours}h old | $perf"; exit 2; fi
if [ "$hours" -ge "$warn_h" ]; then echo "WARNING: newest backup is ${hours}h old | $perf"; exit 1; fi
echo "OK: newest backup is ${hours}h old | $perf"
exit 0

Two habits make checks like this trustworthy. Exit 3 whenever the script itself fails (a missing directory, a refused connection to an API), never 0, so a broken check cannot pass as a healthy service. And keep a timeout on everything the script calls, because the monitor will kill a check that hangs and you lose the reason. Writing your own collectors in more depth is covered in extending the monitor.

Passive checks and pushed results

Some results are better pushed than pulled: a nightly job knows when it finished, and a host behind NAT cannot be reached by the poller. Each tool has its own way in.

printf '[%s] PROCESS_SERVICE_CHECK_RESULT;hq-vmh04;nightly-backup;0;OK: finished in 42 min\n' "$(date +%s)" > /usr/local/nagios/var/rw/nagios.cmd
zabbix_sender -z 10.10.0.50 -s hq-vmh04 -k backup.age -o 3600
echo 'backup_age_seconds 3600' > /var/lib/node_exporter/textfile/backup.prom.tmp && mv /var/lib/node_exporter/textfile/backup.prom.tmp /var/lib/node_exporter/textfile/backup.prom

The first line writes to the Nagios external command file (its path depends on how Nagios was installed). The second sends a value to a Zabbix trapper item. The third uses node_exporter's textfile collector, writing to a temporary file and renaming it so Prometheus never scrapes half a file.

Every passive result needs a freshness rule, or a job that silently stops running looks fine forever. Nagios has check_freshness with a freshness_threshold, Zabbix has the nodata() trigger function, and in Prometheus you can compare time() with node_textfile_mtime_seconds.

Inbound webhooks

Cloud services, CI systems and many appliances can call a URL when something happens. Treat an inbound webhook endpoint as a public API:

• Require a shared secret in a header, or better, verify an HMAC signature of the body.
• Accept requests only from the sender's published address ranges where it has them.
• Answer quickly and queue the work; senders retry on slow responses and you get duplicates.
• Make processing idempotent: the same event ID twice is still one event.

A test call to make sure the mapping works:

curl -sS -X POST https://nms.example.net/hooks/inbound -H 'Content-Type: application/json' -H "X-Hook-Token: $HOOK_TOKEN" -d '{"host":"br2-fw01","object":"wan1","state":"down","message":"ISP circuit down"}'

Security rules for external programs

Anything that runs code on your monitoring server is part of its attack surface.

• Run handlers and checks as a dedicated unprivileged user, never as root.
• Treat input as hostile. Anyone who can send a UDP packet to port 162 or 514 can put any text they like into a trap or a syslog message. Never pass it through a shell (eval, sh -c, backticks), and never build SQL by joining strings.
• Restrict ports 162 and 514 with a firewall to the device ranges that should send them.
• Keep secrets off command lines, where ps shows them to every user; read them from a file with mode 0600.
• If a check really needs root, allow exactly that command in sudoers and nothing more, for example nagios ALL=(root) NOPASSWD: /usr/sbin/smartctl -H /dev/sda.
• Put a timeout on every handler and check, and log what each one did.

Before wiring in a new input, confirm that you know which event fields it fills, what happens when it stops arriving, and who can send it. If any of the three answers is "not sure", it is not ready yet.