Common monitoring errors and how to fix them

Most monitoring problems produce the same dozen messages, and most of those have the same handful of causes. This handbook lists them as symptoms: what you see, why it usually happens, how to check, and what fixes it. The commands use net-snmp, RRDtool and standard Linux tools, so they apply whichever monitoring package sits on top.

One rule before any of it: reproduce the problem by hand, as the same user, from the same machine the poller runs on. A check that works from your laptop proves nothing about the poller, and a script that works in your shell proves nothing about cron. If your tool has a debug or single-host poll mode, use it; otherwise run the underlying command yourself.

sudo -u nms -H snmpget -v2c -c n0c-r34d -t 3 -r 0 10.30.1.11 SNMPv2-MIB::sysUpTime.0

-t 3 waits three seconds and -r 0 disables retries, so a failure shows up quickly instead of after a minute of silent retrying.

1. SNMP "Timeout: No Response from" the device

The agent never answered. SNMP agents drop requests they will not serve without sending any error back, so a wrong community looks exactly like a dead device.

A variant: single gets work but walks time out halfway. Large bulk responses over a VPN can exceed the path MTU and get dropped as fragments. Ask for fewer rows per request with snmpbulkwalk -Cr10, or lower the bulk size in your poller's settings.

2. "No Such Object" and "No Such Instance"

SNMPv2-MIB::sysUpTime = No Such Instance currently exists at this OID
IF-MIB::ifHCInOctets.47 = No Such Instance currently exists at this OID
UCD-SNMP-MIB::laLoad.1 = No Such Object available on this agent at this OID

The agent answered, but not with data. "No Such Instance" means the object exists and that particular row does not. "No Such Object" means the agent does not implement the object at all, or will not show it to you. With SNMPv1 the same situation comes back as an error packet with the reason noSuchName.

Causes, most common first:

• A scalar without its .0. Ask for sysUpTime.0, not sysUpTime.
• A stale index. Interface indexes can change after a reboot, a firmware upgrade or a new line card. Walk IF-MIB::ifName again and compare. On Cisco IOS, snmp-server ifindex persist keeps them stable.
• A 64-bit counter requested over SNMPv1, which cannot carry Counter64 values. Use v2c or v3.
• A restricted view. Some distribution packages ship snmpd.conf with a view that exposes little more than the system group. If a full walk of .1.3.6.1.2.1 stops after a few dozen lines, that is the cause; widen the view for the monitoring community or user.
• The object really is not there: an old agent, or a vendor that never implemented that MIB.

3. "Unknown Object Identifier" and "Cannot find module"

MIB search path: /home/nms/.snmp/mibs:/usr/share/snmp/mibs
Cannot find module (CISCO-PROCESS-MIB): At line 0 in (none)
CISCO-PROCESS-MIB::cpmCPUTotal5minRev: Unknown Object Identifier

This one never reaches the network. The net-snmp tools could not translate a name into a numeric OID because the MIB file, or a MIB it imports, is not on the search path. Copy the vendor's MIB files (and their dependencies, such as CISCO-SMI and CISCO-TC) into ~/.snmp/mibs or /usr/share/snmp/mibs, then load them with -m +CISCO-PROCESS-MIB, or -m ALL while testing. On Debian and Ubuntu, the snmp-mibs-downloader package fetches the standard MIBs, and the mibs : line in /etc/snmp/snmp.conf must be commented out before any are loaded.

Check a translation directly with snmptranslate -On CISCO-PROCESS-MIB::cpmCPUTotal5minRev. For the poller's own configuration, prefer numeric OIDs: they work with no MIB files at all, which removes this whole class of error from production.

4. SNMPv3 "Authentication failure" or "Unknown user name"

snmpget: Authentication failure (incorrect password, community or key)
snmpget: Unknown user name

v3 at least tells you something. "Unknown user name" means the user does not exist on the agent, or was created for a different engine ID: v3 keys are tied to the agent's engine ID, so if a device is re-imaged or its engine ID changes, users have to be created again. "Authentication failure" means a wrong passphrase or the wrong algorithm (SHA against MD5, for example). A wrong privacy passphrase or algorithm (AES against DES) often shows up as a timeout or a decryption error instead, so test -l authNoPriv first and add privacy once that works.

5. Counters that go backwards, or graphs with huge spikes

A traffic graph shows a spike of 40 Gbit/s on a 1 Gbit/s link, or a rate below zero.

• 32-bit wrap. A Counter32 tops out at 4,294,967,295 and starts again from zero. At 1 Gbit/s an octet counter wraps in about 34 seconds; at 100 Mbit/s, in under six minutes. With a five-minute poll you cannot tell how many times it wrapped. Use ifHCInOctets and ifHCOutOctets, as explained in interface counters explained.
• A device reboot or a manual clear counters. Compare sysUpTime.0 between polls: if it went down, the counters were reset.
• Polling a virtual address. A VRRP or HSRP address can be answered by either router, so successive polls read counters from two different devices. Poll each router's real address.
• A moved interface index (see entry 2).

The fixes: use 64-bit counters, cap each RRD data source at the link's real maximum so impossible values are stored as unknown, and use a DERIVE data source with a minimum of 0 if you prefer a reset to show as a gap. For a 1 Gbit/s port stored in bytes per second:

rrdtool tune hq-core-sw01_port3.rrd --maximum traffic_in:125000000 --maximum traffic_out:125000000

The counter rate calculator does the arithmetic for other speeds and poll intervals.

6. Graphs with gaps

RRDtool stores "unknown" when updates arrive further apart than a data source's heartbeat. Gaps therefore mean late or missing updates.

rrdtool info hq-core-sw01_port3.rrd | grep -E '^step|minimal_heartbeat|^last_update'
rrdtool lastupdate hq-core-sw01_port3.rrd
chronyc tracking

Common causes: the poller takes longer than its interval (time a full cycle; if a five-minute cycle takes six minutes, some hosts are always late), devices that time out and drag the cycle out, a heartbeat set equal to the step so one late update leaves a hole, and clocks that drift or jump on the poller. Fixes: more poller processes or a staggered schedule, a heartbeat of about twice the step (rrdtool tune FILE --heartbeat traffic_in:600 for a 300-second step), and working NTP. Polling covers how to tell whether a poller is falling behind.

7. RRD "illegal attempt to update using time"

ERROR: illegal attempt to update using time 1791968400 when last update time is 1791968400 (minimum one second step)

RRDtool refuses any update that is not later than the previous one. Either two processes are writing the same file (an overlapping cron run, a second poller node, a manual test during a scheduled poll), or the clock went backwards. Find the second writer with ps aux | grep -i poll during a poll window, then make overlapping runs impossible with a lock:

*/5 * * * * nms flock -n /run/lock/nms-poller.lock /opt/nms/bin/poller

If the clock is to blame, configure chrony to slew rather than step after startup, and check that every poller node syncs to the same sources.

8. Works by hand, fails from cron

Cron runs jobs with a minimal environment: a short PATH, no login profile, often a different HOME, and a working directory you did not choose. A different HOME matters more than it looks, because net-snmp reads ~/.snmp/snmp.conf and its MIB directory from there.

Check where the job is defined (crontab -l -u nms, files in /etc/cron.d, which need a user field) and what cron logged (journalctl -u cron, or crond on Red Hat family systems). Reproduce cron's conditions with sudo -u nms env -i PATH=/usr/bin:/bin HOME=/home/nms /opt/nms/bin/poller. Fix with absolute paths, a PATH= line at the top of the crontab, and one owner for every scheduled job. A job that once ran as root also leaves root-owned files behind, which is the next entry.

9. "Permission denied" on logs, graphs or spool files

ERROR: opening '/var/lib/nms/rrd/br2-fw01_wan1.rrd': Permission denied

Find files the poller cannot own with find /var/lib/nms -not -user nms -ls. Remember the web server usually draws the graphs, so its user needs read access too; a shared group with the setgid bit on directories keeps new files consistent. On SELinux systems, check ls -Z and ausearch -m avc -ts recent, and run restorecon -Rv on the data directory. Rule out a full disk while you are there: df -h, and df -i, because thousands of small RRD files can exhaust inodes long before space.

10. Database locks and deadlocks

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
database is locked

The first two are MySQL or MariaDB, the third SQLite. The usual trigger is housekeeping: a purge that deletes months of events in one transaction while pollers are writing, or a backup that locks tables. Look with SHOW FULL PROCESSLIST; and the latest deadlock section of SHOW ENGINE INNODB STATUS\G; on PostgreSQL, query pg_stat_activity for sessions whose wait_event_type is Lock. Delete old rows in small batches, schedule purges away from busy polling periods, index the timestamp columns that purges filter on, and dump InnoDB tables with mysqldump --single-transaction. Housekeeping schedules are covered under administration.

11. Email actions that never arrive

Send a test with echo "test from $(hostname)" | mail -s "monitor mail test" oncall@example.com, then schedule a daily heartbeat alert and treat its absence as an incident.

When nothing here matches

Capture the exact message, the command that produced it, the tool and version, and what changed recently, then work through the help page flow. The net-snmp and RRDtool documentation explain every flag used above. Whatever fixes it, add it to your own runbook: the second occurrence should take five minutes.