Writing a custom check or poller, from plan to alert

Every monitoring system ships with checks for the usual suspects: interfaces, CPU, disks, ping. Sooner or later you meet something it does not know about. A UPS with an odd MIB, a line-of-business app with a JSON status page, a storage shelf that only tells you its health through a CLI. You can wait for someone else to write a template, or you can write the check yourself in an afternoon. This guide is the method, independent of which tool you run, in six steps.

Step 1: plan before you write code

Most rewrites of custom checks come from skipping this step. Fill in a short worksheet first, even if it lives in a text file next to the script.

Three items deserve extra thought.

Discovery should fail fast

Find the one cheap query that proves the object does not exist on a host, and stop there. Walking a large vendor table on every device of every type, every discovery run, is how discovery jobs end up taking all night.

The index must be stable

If it can change, the history of one object gets attached to another. Interface indexes are the classic case: some devices renumber ifIndex after a reboot or a line card swap, which is why many tools key interfaces on ifName or ifDescr and look up the index each cycle. A TCP port number is an ideal index; the order in which a scan found things is the worst one.

Names and descriptions are different fields

The device's own name (GigabitEthernet1/0/24) identifies the object and should match what appears in its traps and syslog lines, so events land on the right object. The description ("uplink to building C") is for humans. Never let a check guess a description from a neighboring row: a wrong description on an important object is worse than none.

Step 2: find the data

SNMP

Walk the device and see what it offers. Start broad, then narrow:

snmpwalk -v2c -c n0c-r34d 192.0.2.30 1.3.6.1.2.1 | less
snmpwalk -v2c -c n0c-r34d -M +$HOME/mibs -m ALL 192.0.2.30 enterprises | less

The second line loads every MIB in your search path, including a vendor MIB you dropped into ~/mibs, so the private part of the tree prints with names instead of bare numbers. Once you have a candidate, snmptranslate tells you what it is:

snmptranslate -On IF-MIB::ifHCInOctets      # numeric form: .1.3.6.1.2.1.31.1.1.1.6
snmptranslate -Td IF-MIB::ifHCInOctets      # syntax, access, description, UNITS
snmptranslate -Tp IF-MIB::ifXTable          # the subtree as a tree

Read the -Td output carefully. The SYNTAX line says whether it is a Counter32, Counter64, Gauge32, INTEGER with named states, or TimeTicks. A UNITS or DISPLAY-HINT clause tells you whether "275" means 27.5 degrees. The net-snmp documentation covers the output and MIB loading options in detail.

HTTP and CLI

Many applications expose a status page. Look at it by hand before you script it:

curl -s -m 5 http://192.0.2.40:8080/health | jq .

For a CLI, run the command as the user your monitor runs as, not as root, and save a few real outputs (healthy, degraded, broken) to test your parser against. Parsing human-oriented output is fragile; prefer a machine format (--json, --format csv) if the tool has one.

Step 3: write the collector

Keep collectors small, read-only and quick. They should print what they found and exit with a status code. The convention most tools understand comes from Nagios plugins: exit 0 for OK, 1 for warning, 2 for critical, 3 for unknown, one line of text, and optional performance data after a | character. The guide to external programs covers how different tools call scripts like this.

Here is a complete check for a job queue that reports its depth through a JSON status page:

#!/usr/bin/env python3
"""check_queue.py: read a job queue depth from an HTTP status endpoint."""
import json
import sys
import urllib.request

URL = "http://192.0.2.40:8080/health"
WARN, CRIT = 200, 500

try:
    with urllib.request.urlopen(URL, timeout=5) as resp:
        data = json.load(resp)
    depth = int(data["queue_depth"])
    workers = int(data["workers"])
except Exception as exc:  # network error, bad JSON, missing key
    print(f"UNKNOWN: cannot read {URL}: {exc}")
    sys.exit(3)

state, code = "OK", 0
if depth >= CRIT:
    state, code = "CRITICAL", 2
elif depth >= WARN:
    state, code = "WARNING", 1

print(f"{state}: queue depth {depth}, {workers} workers"
      f" | queue_depth={depth};{WARN};{CRIT};0 workers={workers};;;0")
sys.exit(code)

Run it by hand, then run it as the monitoring user, then point it at an address with nothing listening and confirm it says UNKNOWN rather than crashing with a traceback.

Step 4: store and graph it

How you store a value depends on what kind of value it is. Getting this wrong is the most common mistake in custom checks.

Counters only make sense as a difference between two readings divided by the time between them, and they wrap or reset. A 32-bit octet counter on a busy 10 Gb/s port can wrap in a few seconds, which is why ifHCInOctets exists; the article on interface counters explains the wrap in detail. Gauges are stored as they are. Normalize units at collection time (octets to bits, tenths of a degree to degrees) and write the unit into the data source name or label so nobody has to guess in two years.

Step 5: thresholds and an alert

Pick thresholds from data, not from a vendor brochure. Collect for a week or two, look at the busy-hour peaks, then set a warning level that a normal week never reaches and a critical level that means someone should act today. Two habits keep the alert quiet when it should be:

• Require persistence: fire after the condition holds for several cycles (three polls in a row, or a for: 15m clause in Prometheus-style rules), not on a single sample.
• Use hysteresis: clear the alarm at a lower level than the one that raised it, so a value hovering at the line does not flap.

Connect the new check to the host's existing dependencies, so a dead switch upstream does not produce a page for every queue behind it.

Step 6: share it

A check that only works on your monitoring server is a liability when you leave. Package it the way your tool expects (a Zabbix template, a Nagios or Icinga plugin with a command definition, a LibreNMS discovery definition, a Prometheus exporter module), keep it in version control, and write a short header that lists the MIB or endpoint it reads, the units, sample output, and why the thresholds are what they are.

Pitfalls checklist

• Treating a counter as a gauge (graphs that only ever climb) or a gauge as a counter (rates that are nonsense).
• Wrong units, especially octets versus bits and scaled integers.
• A collector timeout longer than the poll interval, so slow runs pile up; keep it well under the scheduler's own timeout too.
• Running as root because it was easier. Give the monitoring user exactly the access it needs, and if a command truly needs privilege, a sudo rule for that single command.
• Locale surprises: a decimal comma from a CLI parsed as two numbers.
• Checks with side effects. A check must never restart, clear or change anything.
• No test against failure. Unplug it, block it, feed it garbage, and make sure the check reports something sensible.

For a full, start-to-finish example of these steps with a real counter, follow the ICMP worked example. For how the finished check fits into poll scheduling, see polling.