How often should you poll? Choosing a polling interval

Every new monitoring install reaches the same argument within a week. The default says five minutes. Someone on the team wants ten seconds because a vendor demo looked nice at ten seconds. Someone else points out that the core switch CPU is already at 40 percent. All three have a point, and the answer is not one number. It is a small set of numbers, chosen per metric, with a few hard limits you cannot argue with.

Where five minutes came from

The five-minute interval comes from MRTG, the tool that taught a generation of network engineers to graph router traffic. MRTG ran every five minutes, usually from cron, read the octet counters on each interface and drew four graphs: a daily graph of 5-minute averages, a weekly one of 30-minute averages, a monthly one of 2-hour averages and a yearly one of daily averages. When RRDtool took over the storage side, its default step was 300 seconds, and that number spread into Cacti, LibreNMS and most of the tools built on round-robin databases. Both Cacti and LibreNMS still default to five minutes at the time of writing.

It was a sensible choice for the time. Each run started a fresh process and walked every router in turn on modest hardware. Links were T1s and 10 Mbit/s Ethernet, where a 32-bit octet counter takes 57 minutes to wrap at full speed, so five minutes left plenty of margin. And for the question people were asking, "how full is this link over a day", a 5-minute average is a perfectly good answer.

The trouble is that the question changed and the default did not.

What a five-minute average hides

Counters do not lose bytes. The difference between two readings of ifHCInOctets is exactly how many bytes crossed the port in that window, so the average is exact. What the average loses is shape.

Take a 1 Gbit/s uplink that runs at 100 percent for 30 seconds and at 10 percent for the other 270 seconds of a polling window. The window carried 30 × 1,000 + 270 × 100 = 57,000 megabits, and 57,000 / 300 gives 190 Mbit/s. The graph shows a link at 19 percent. Users saw 30 seconds of drops and a stalled backup job. Poll the same link every 60 seconds and the minute containing the burst shows (30 × 1,000 + 30 × 100) / 60 = 550 Mbit/s, which at least looks like something happened.

Gauges behave worse. CPU load, temperature and queue depth are read as a single value at the moment of the poll. A 40-second CPU spike that starts and ends between two 5-minute polls does not appear in the graph at all, not even diluted.

A Nyquist-style rule of thumb

Sampling theory says you need to sample a signal at more than twice its highest frequency to reconstruct it. Monitoring is not signal reconstruction, but a rough version of the rule carries over cleanly: poll at an interval no longer than half the shortest event you care about.

The reasoning is simple. If an event lasts T seconds and your interval is T/2 or shorter, at least one complete polling interval falls inside the event. For a counter, that interval shows the event at full height instead of smeared across two half-height samples. For a gauge, it means at least one poll lands inside the event and you see it at all. So if two minutes of saturation on a WAN link is something you need to know about, poll it every minute. If you only care about saturation that lasts a quarter of an hour, five minutes is fine.

Counter wrap sets a hard floor

Rates come from the difference between two counter readings, and a 32-bit counter rolls over at 2^32, which is 4,294,967,296 bytes (about 4.29 GB). A poller can correct for one wrap between readings. It cannot detect two, and it has no way to know how many happened. The time to wrap at full line rate:

At 1 Gbit/s you would need to poll comfortably faster than every 34 seconds, with poll jitter on top, just to keep 32-bit counters honest. At 10 Gbit/s it is hopeless. Packet counters are not safe either: at 10 Gbit/s with minimum-size frames a port moves about 14.88 million packets per second, which wraps a 32-bit packet counter in about 289 seconds, inside a 5-minute poll. The fix is the high-capacity columns (ifHCInOctets, ifHCOutOctets and the ifHC...Pkts family), which need SNMP v2c or v3 as explained in the SNMP versions article. The counter rate calculator works out wrap time and rate for any speed and interval.

So the first rule is not about intervals at all: on any link of 1 Gbit/s or faster, poll 64-bit counters, or the interval question is moot.

What faster polling costs

Three things get more expensive when you shorten the interval, and only one of them is disk.

Device CPU

The SNMP agent on a switch or router runs on the control-plane CPU, which on many access switches is a modest embedded part that also runs spanning tree and routing protocols. Reading a handful of counter columns is cheap. Walking the MAC address table, the ARP table or a full routing table is not, and doing it every minute is how you end up explaining CPU alarms caused by your own monitoring.

Poller capacity

Going from 5 minutes to 1 minute is five times the requests; going to 15 seconds is twenty times. The whole poll cycle has to finish inside the interval. If walking 1,000 devices takes four minutes today, a 1-minute interval needs more parallel workers or a second poller, not a config change.

Storage

Storage is the easiest to quantify. Take 1,000 interfaces with four series each (octets in and out, errors and discards), so 4,000 series. Prometheus documents roughly 1 to 2 bytes per sample after compression; I used 2:

Round-robin databases are fixed in size instead: 8 bytes per data source per row. A year at 1-minute resolution is 525,600 rows, about 4.2 MB per data source, against about 841 KB at 5 minutes. That is why RRD setups keep fine resolution for a few days and consolidate older data into coarser archives.

None of these numbers is scary. Disk is the cheapest part of faster polling. Device CPU and poller time are the real limits.

Detection time is the number people forget

Most alert rules wait for several bad polls in a row so a single blip does not page anyone. With a 5-minute interval and a three-polls rule, the alert fires 10 to 15 minutes after the problem starts, depending on where in the cycle it began. At 1 minute, the same rule fires in 2 to 3 minutes. If your outage reports say "monitoring noticed 14 minutes after users did", the interval is usually why, and the triggers guide has more on tuning the rule side.

One interval does not fit every metric

This is where I end up on most networks. Treat it as a starting point and adjust it to your device CPUs and your poller.

Two notes on the table. Errors and discards have no 64-bit versions in IF-MIB, but they count packets with problems, so they rarely climb fast enough to wrap between polls. And the 1-minute rows are where device CPU complaints come from first, so roll them out to a few devices and watch the CPU graph before you apply them everywhere.

Setting it up without regrets

A few habits make any interval work better:

1. Start with 5 minutes as the default and promote specific things to 1 minute: uplinks, WAN links, the core and router CPUs. It is easier to justify adding load than to explain why you took monitoring down a notch.
2. Compute rates from the timestamps you actually recorded, not the nominal interval. A poll that arrives at 63 seconds instead of 60 overstates the rate by 5 percent if you divide by 60.
3. Keep raw samples at the fine interval for 7 to 30 days and consolidate after that. Nobody needs last February at 1-minute resolution.
4. Watch the poller's own cycle time, and alert when a cycle takes more than about 80 percent of the interval. Overrunning pollers produce gaps that look like outages.
5. Write the chosen intervals down next to the polling configuration, so the next person knows they were chosen and not inherited. The polling guide walks through the poller side of the setup.

If you keep one thing from this: poll each metric at half the length of the shortest event you need to see, and never longer than its counters allow.