Worked example: graphing incoming ICMP on a Linux server

The guide to writing custom checks describes the method in general terms. This page walks through one complete example, from finding an OID to an alert, so you have a template to copy for your own pollers. The goal: a graph of ICMP messages arriving at a Linux server, with a separate line for ping requests, and a warning when the ping rate stays unusually high.

ICMP is a good first poller. Every IP host keeps ICMP counters, the counters are exposed over SNMP on any Linux box running net-snmp, and you can make them move on demand by pinging the server from another machine. ICMP also carries more than ping: destination unreachable, time exceeded and the "fragmentation needed" messages that path MTU discovery depends on all count as incoming ICMP, so the graph sometimes tells you about network trouble elsewhere.

Step 1: pick the OIDs

The counters live in the icmp group of MIB-II, defined in RFC 1213 and carried forward in IP-MIB. Later revisions of IP-MIB (RFC 4293) mark these scalars deprecated in favor of per-IP-version tables, but net-snmp on Linux still answers them and nearly every tool still polls them. We will use three:

The trailing .0 is the instance: these are scalars, one value per host. Confirm the names and numbers on your own machine rather than trusting a web page:

$ snmptranslate -On IP-MIB::icmpInEchos.0
.1.3.6.1.2.1.5.8.0
$ snmptranslate -Td IP-MIB::icmpInMsgs | grep -E 'SYNTAX|MAX-ACCESS'
  SYNTAX        Counter32
  MAX-ACCESS    read-only

Counter32 is the important word. It tells you the value only ever climbs (until it wraps or resets), so the useful number is the rate, not the value.

Step 2: let the agent answer

On Debian and Ubuntu the packaged snmpd.conf listens only on localhost and exposes only a small "systemonly" view, so a poller on another host gets a timeout or an empty reply. In /etc/snmp/snmpd.conf, change the existing agentaddress line and add a view and a read-only community limited to your poller's address (192.0.2.10 here):

agentaddress  udp:161
view          monview  included  .1.3.6.1.2.1.1
view          monview  included  .1.3.6.1.2.1.5
rocommunity   n0c-r34d  192.0.2.10  -V monview

sudo systemctl restart snmpd
sudo ufw allow from 192.0.2.10 to any port 161 proto udp

A community string travels in clear text. On anything beyond a lab network, use SNMPv3 with authentication and privacy instead: net-snmp ships net-snmp-create-v3-user to create a read-only user, which you then limit to the same view. The command lines below change only in their credentials:

snmpget -v3 -l authPriv -u icmpmon -a SHA -A 'long-auth-passphrase' \
  -x AES -X 'long-priv-passphrase' 192.0.2.20 IP-MIB::icmpInMsgs.0

Step 3: check by hand

From the poller:

$ snmpget -v2c -c n0c-r34d 192.0.2.20 \
    1.3.6.1.2.1.5.1.0 1.3.6.1.2.1.5.8.0 1.3.6.1.2.1.5.22.0
IP-MIB::icmpInMsgs.0 = Counter32: 48211
IP-MIB::icmpInEchos.0 = Counter32: 40917
IP-MIB::icmpOutEchoReps.0 = Counter32: 40917

Now ping the server 20 times from a third machine (ping -c 20 192.0.2.20) and run snmpget again. All three numbers should rise by 20, give or take whatever else arrived meanwhile. On the server itself, grep '^Icmp:' /proc/net/snmp prints the kernel's own counters, which is where net-snmp gets these values on Linux; they count from boot.

Step 4: from counter to rate

A rate is the difference between two readings divided by the seconds between them. If icmpInMsgs reads 48,211 at 10:00:00 and 48,511 at 10:05:00, the server received 300 messages in 300 seconds: 1.0 per second. The counter rate calculator does the same arithmetic if you want to check a reading by hand.

Two things break the simple subtraction. A Counter32 wraps to zero after 4,294,967,295, which an ICMP counter on a quiet server will not reach for years, though a busy one might. And the counter starts again from zero when the host reboots. Both show up as a negative difference. A tool that treats every negative difference as a wrap turns each reboot into a huge false spike; one that discards negative differences loses a single sample at a wrap. For a counter that rarely wraps but resets at every reboot, discarding is the better trade, and it is what the storage below does.

Step 5: create the RRD

RRDtool stores the data in a fixed-size file and does the rate maths for us. One file, three data sources, a five-minute step:

rrdtool create /var/lib/monitoring/rrd/icmp-web01.rrd \
  --step 300 \
  DS:icmpin:DERIVE:600:0:U \
  DS:echoin:DERIVE:600:0:U \
  DS:echoreply:DERIVE:600:0:U \
  RRA:AVERAGE:0.5:1:2016 \
  RRA:AVERAGE:0.5:12:2160 \
  RRA:MAX:0.5:12:2160 \
  RRA:AVERAGE:0.5:288:730 \
  RRA:MAX:0.5:288:730

Reading the pieces: DERIVE turns each update into a per-second rate, and the minimum of 0 throws away the negative rate a reboot would create (a COUNTER data source would instead assume a wrap). The heartbeat of 600 seconds means two missed polls in a row become "unknown" rather than a guess. The archives keep 5-minute averages for 7 days (2,016 rows), hourly averages and maxima for 90 days, and daily averages and maxima for two years.

Step 6: poll every five minutes

A small script, saved as /usr/local/bin/poll-icmp.sh, does the polling and the update:

#!/bin/sh
HOST=192.0.2.20
COMMUNITY=n0c-r34d
RRD=/var/lib/monitoring/rrd/icmp-web01.rrd

VALUES=$(snmpget -v2c -c "$COMMUNITY" -t 2 -r 1 -Oqv "$HOST" \
    1.3.6.1.2.1.5.1.0 1.3.6.1.2.1.5.8.0 1.3.6.1.2.1.5.22.0) || exit 1

set -- $VALUES
if [ $# -ne 3 ]; then
    logger -t poll-icmp "unexpected reply from $HOST: $VALUES"
    exit 1
fi

rrdtool update "$RRD" --template icmpin:echoin:echoreply "N:$1:$2:$3"

-Oqv makes snmpget print bare values, one per line. If an object is missing, the reply turns into several words and the argument count check catches it. Schedule it with a file in /etc/cron.d:

*/5 * * * * monitor /usr/local/bin/poll-icmp.sh

Step 7: draw the graph

rrdtool graph /var/www/graphs/icmp-web01-day.png \
  --start -1d --end now \
  --title "ICMP received, web01" \
  --vertical-label "messages per second" \
  --width 720 --height 220 --lower-limit 0 \
  DEF:in=/var/lib/monitoring/rrd/icmp-web01.rrd:icmpin:AVERAGE \
  DEF:echo=/var/lib/monitoring/rrd/icmp-web01.rrd:echoin:AVERAGE \
  VDEF:inavg=in,AVERAGE \
  VDEF:inmax=in,MAXIMUM \
  AREA:in#7f9fbf:"All ICMP in" \
  LINE2:echo#1f3b57:"Echo requests" \
  GPRINT:inavg:"avg %6.2lf/s" \
  GPRINT:inmax:"peak %6.2lf/s\l"

Change --start -1d to -1w or -1y for longer views; RRDtool picks the archive that fits. Remember that each point is an average: a 20-second burst of pings inside a 5-minute step shows up as a modest bump, not a spike. If you also graph echoreply, the line normally sits right on top of the echo requests. When they part, the host is receiving pings and not answering all of them, so look at its firewall and ICMP settings.

Step 8: alert on a ping flood

Collect for a week or two before choosing numbers. A web server pinged by a couple of monitors sees a steady trickle; a flood is a sustained rate many times the normal busy-hour figure. The thresholds below (20 and 50 echo requests per second over 15 minutes) are placeholders for yours. This check-icmp-flood.sh follows the usual plugin convention, so most monitors can run it as a check:

#!/bin/sh
RRD=/var/lib/monitoring/rrd/icmp-web01.rrd
WARN=20
CRIT=50

RATE=$(rrdtool graph /dev/null --start -15min --end now \
    DEF:e="$RRD":echoin:AVERAGE VDEF:avg=e,AVERAGE PRINT:avg:%.1lf | tail -n 1)

case "$RATE" in
    ""|*nan*) echo "UNKNOWN: no recent data in $RRD"; exit 3 ;;
esac

PERF="echo_rate=$RATE;$WARN;$CRIT;0"
if awk -v r="$RATE" -v c="$CRIT" 'BEGIN { exit !(r >= c) }'; then
    echo "CRITICAL: $RATE echo requests/s | $PERF"; exit 2
elif awk -v r="$RATE" -v w="$WARN" 'BEGIN { exit !(r >= w) }'; then
    echo "WARNING: $RATE echo requests/s | $PERF"; exit 1
fi
echo "OK: $RATE echo requests/s | $PERF"; exit 0

Route the warning to a chat channel and the critical to whoever owns the server; triggers and actions covers how to wire that up without paging people at 3 a.m. for a misconfigured monitoring probe. Keep in mind what this graph does not tell you: a server that answers ping can still be failing at everything users care about, a point the article ping is not monitoring makes at length.

The same poller in other tools

Nothing here is specific to shell scripts. In Zabbix, the poll becomes an item of type SNMP agent with 1.3.6.1.2.1.5.8.0 as its OID and a "Change per second" preprocessing step, and a trigger such as avg(/web01/icmp.in.echos,15m)>50 replaces the check script. With Prometheus, add the three objects to a module in the snmp_exporter generator configuration, regenerate, and scrape; the exporter exposes Counter32 values as counters, so rate(icmpInEchos[5m]) gives the per-second figure and an alerting rule with for: 15m does the threshold work. Cacti, LibreNMS and Nagios-family tools follow the same pattern under different names: an OID, a counter type, a rate, a threshold. Once you have built this one by hand with net-snmp and RRDtool, those screens read as shortcuts for steps you already understand.