Polling explained: pollers, intervals, timeouts and keeping up

A poll is a question with a deadline. The monitor asks a device something specific, waits a set time for the answer, records what it got (or that it got nothing), and moves on to the next question. Run that loop for every object on every host, on a schedule, and you have the pull half of network monitoring. This chapter covers what kinds of questions a poller can ask, how often to ask them, what to do when nobody answers, and how to tell when the poller can no longer keep up.

Five kinds of poller

Each of them can be tried by hand, which is the fastest way to find out whether a problem is in the device or in the monitor's configuration. The examples use the net-snmp command-line tools from the net-snmp project:

snmpget -v2c -c n0c-r34d 10.20.4.11 1.3.6.1.2.1.1.3.0 1.3.6.1.2.1.31.1.1.1.6.48
snmpwalk -v2c -c n0c-r34d 10.20.4.11 1.3.6.1.2.1.31.1.1.1.1
fping -c 5 -q 10.20.4.11 10.20.4.12
nc -zv -w 3 10.20.5.10 443
curl -s -o /dev/null -w '%{http_code} %{time_total}\n' https://intranet.example.com/health

The first line fetches sysUpTime and the 64-bit inbound octet counter (ifHCInOctets) for interface index 48 in a single request. The second walks ifName, so you can see which index belongs to which port. A community string is used here for brevity; on any network you do not fully trust, use SNMPv3 with authentication and privacy, or at least an access list that only answers the monitor.

Scripts follow their own conventions (the widely copied plugin convention is exit code 0 for OK, 1 for warning, 2 for critical and 3 for unknown). Writing one well is covered in the chapter on writing a custom check.

How often to ask

The interval is a trade between detail and cost.

Averaging hides more than people expect. If a link runs at 100 percent for 30 seconds and at 10 percent for the other 270, its 5-minute average reads 19 percent. Users felt that half minute; the graph never will. That is why many setups poll status and core uplinks every minute and leave the bulk of access ports at five.

Counters add a hard limit. A 32-bit octet counter (ifInOctets) wraps at 2^32, which on a link carrying a steady 1 Gbit/s happens about every 34 seconds, and on a 100 Mbit/s link about every 5.7 minutes. Poll less often than the wrap time and you cannot tell one wrap from two. The fix is the 64-bit ifHCInOctets (1.3.6.1.2.1.31.1.1.1.6), available over SNMPv2c and v3, which would take decades to wrap even at 100 Gbit/s. To check a rate by hand, take two readings and their timestamps and use the counter rate calculator.

The full argument for picking intervals per object type is in how often should you poll.

Timeouts and retries

The net-snmp tools default to a 1-second timeout and 5 retries, which suits a person at a prompt better than a poller. With those defaults, every request to a silent device ties up a worker for six seconds or more. A poller that asks a dead switch for ten tables one after another can lose a minute on it.

Two habits help. First, check reachability before the expensive work: if the device fails a ping or a single sysUpTime GET, mark it down and skip the rest of its poll. Most tools do this already. Second, set timeouts and retries per poller type, deliberately:

snmpget -v2c -c n0c-r34d -t 2 -r 1 10.20.4.11 1.3.6.1.2.1.1.3.0

Two seconds and one retry is a reasonable starting point on a LAN. Raise the timeout for devices across a WAN or a satellite link, not the retry count.

Keep two kinds of retry apart. SNMP retries resend a single UDP request that got no answer. Check retries (often called "attempts" or "max check attempts") repeat a whole check before the state is allowed to change. The first fights packet loss; the second fights false alarms. SNMPv3 adds one wrinkle: the first request to a device includes discovery of the agent's engine ID, so a poller's first contact with each device costs an extra round trip.

Staggering the schedule

A cron line that starts every host's poll at exactly :00, :05 and :10 sends a burst of thousands of requests in the same second, then nothing for four and a half minutes. The burst lands on the poller's CPU, on the management network, on the graph store's write queue and on any device whose agent shares a slow control-plane CPU.

Spread the starts across the interval. Give each host a fixed offset rather than a random one, so the gap between its samples stays constant, which matters for rate calculations. A small wrapper does it for a home-grown poller:

#!/bin/sh
host="$1"
offset=$(( $(printf '%s' "$host" | cksum | cut -d' ' -f1) % 60 ))
sleep "$offset"
exec /opt/nms/bin/poll "$host"

Most full-size tools schedule this way internally. Prometheus, for example, spreads scrape start times across the scrape interval for each target rather than firing them all at once.

What happens to a result

Each poll result goes to one or more back ends:

• Store it. Counters become rates (difference between two readings divided by the time between them) and go to RRD files or a time-series database along with gauges. If sysUpTime went backwards or ifCounterDiscontinuityTime changed, the device restarted or reset counters, and that sample should be discarded, not graphed as a spike.
• Compare it. A value checked against thresholds produces a state: OK, warning, critical.
• Report a change. A state that differs from the last poll becomes an event, which the alarm machinery takes from there. A state that has not changed produces nothing new, which keeps the event stream readable.

Poller groups and templates

Nobody configures polling object by object. You define, once per object type, which values to collect, how often, with which thresholds and graphs, and attach that definition to every matching object. Tools name the idea differently: templates in Zabbix, discovery and polling modules in LibreNMS, rule sets in Checkmk, scrape jobs and snmp_exporter modules in a Prometheus setup. Older SNMP tools often call it a poller group.

A workable split for a campus network: access ports get status every 60 seconds and traffic, errors and discards every 5 minutes; uplinks and WAN links get everything every 60 seconds; chassis health (CPU, memory, temperature, fans, power supplies) every 5 minutes; storage and certificate expiry every 30.

GETBULK

SNMPv1 could only walk a table one GETNEXT at a time, one value per round trip. SNMPv2c and v3 add GetBulkRequest, defined in RFC 3416, which asks for many rows at once. It takes two parameters: non-repeaters (how many of the requested OIDs to fetch once) and max-repetitions (how many rows to return for each of the rest).

Walking ifHCInOctets on a 52-port switch takes 53 GETNEXT requests (one per port, plus one that runs off the end of the column). With max-repetitions set to 25 it takes 3. Compare them yourself:

time snmpwalk     -v2c -c n0c-r34d 10.20.4.11 1.3.6.1.2.1.31.1.1.1.6 > /dev/null
time snmpbulkwalk -v2c -c n0c-r34d -Cr25 10.20.4.11 1.3.6.1.2.1.31.1.1.1.6 > /dev/null
snmpbulkget -v2c -c n0c-r34d -Cn1 -Cr10 10.20.4.11 1.3.6.1.2.1.1.3 1.3.6.1.2.1.31.1.1.1.6

The last line returns sysUpTime.0 once (the non-repeater) and the first ten inbound counters in one packet. Do not set max-repetitions sky high. When a response would be too large, the agent returns fewer values than asked, and some older or busy agents simply stop answering. Start between 10 and 25 and test.

Is the poller keeping up?

The one number to watch is how long a full poll cycle takes compared with the interval. A 5-minute cycle that finishes in 90 seconds is healthy. One that takes 280 seconds is about to fall over, and one that takes 320 already has, whatever the dashboard says.

Signs that it is behind: gaps or flat stretches in graphs, alarms that show up a cycle late, "last polled" times drifting later, and poller processes still running when the next cycle starts. Cron will happily start a second copy on top of the first; wrap the job with flock -n /run/nms-poll.lock so a late cycle is skipped instead of stacked.

Tools expose the measurement in different places. Zabbix has internal items such as zabbix[process,poller,avg,busy] and a queue view of overdue items. Prometheus records scrape_duration_seconds for every target next to its up metric. Cacti and LibreNMS log each cycle's run time and show poller statistics.

The arithmetic is simple. Say 400 devices take 6 seconds each on average. That is 2,400 worker-seconds per cycle, so a 300-second interval needs 8 workers running flat out; run 16 and the cycle finishes in about 150 seconds with room to spare. If more workers do not help, the bottleneck is elsewhere: dead devices eating timeouts, walks that should be bulk requests, or disk writes on the graph store. Past a certain size, or across slow WAN links, the answer is to split the work across several pollers, which is the subject of distributed polling.

Before you add workers, run through this list:

1. Disable objects nobody looks at: unused ports, loopbacks, virtual interfaces.
2. Use GETBULK and 64-bit counters everywhere the device supports them.
3. Cut SNMP retries to one and make sure dead hosts are skipped early.
4. Move slow-changing values to longer intervals.
5. Stagger start times.
6. Only then add workers, or another poller.