Help: getting unstuck with network monitoring

Monitoring fails in a small number of ways, and most of them look alike from the console: a host turns red, a graph goes flat, an alert never arrives. Before you change any configuration, work out which layer is broken. This page gives you a short troubleshooting flow, the five commands worth running first, a key to the common error messages, and advice on asking for help when you are still stuck. Everything here works with any monitoring tool. We do not offer one-to-one support or remote access, but the steps below solve most problems before anyone else needs to look.

Four places the fault can be

Every check crosses four layers: the monitoring server, the network path, the credentials and access rules, and the device itself. Test them in that order, from the monitor outward, and run every test from the monitoring server. A laptop on another VLAN proves very little about what the poller can reach.

If ten unrelated hosts all went red at 02:00, look at the monitor before you look at ten devices. A full disk, a stuck poller or a locked database produces exactly that picture.

The five commands to run first

Run these on the monitoring server. The examples use a documentation address and a made-up read-only community, n0c-r34d; substitute your own.

1. ping

ping -c 5 192.0.2.10

This tells you whether the device answers ICMP, with loss and round-trip time. A failure does not prove the device is down, because many firewalls drop ICMP and busy devices rate-limit it. A success does not prove the service works either, which is why ping is not monitoring on its own.

2. traceroute or mtr

mtr -n -r -c 20 192.0.2.10
traceroute -n 192.0.2.10

These show where along the path packets stop or start getting lost. mtr sends many rounds, so it catches loss that comes and goes. Loss on a middle hop that does not carry through to the last hop usually means that router answers probes at low priority, not that it drops your traffic.

3. snmpget

snmpget -v2c -c n0c-r34d 192.0.2.10 1.3.6.1.2.1.1.3.0

This asks for one value, sysUpTime. If it comes back, the path, the SNMP version, the community and the device's access list are all fine. For SNMPv3 the same test looks like this:

snmpget -v3 -l authPriv -u monitor -a SHA -A 'auth-passphrase' -x AES -X 'priv-passphrase' 192.0.2.10 1.3.6.1.2.1.1.3.0

An uptime much shorter than you expected is a finding in itself: the device rebooted, and a reboot explains a lot of strange graphs.

4. snmpwalk

snmpwalk -v2c -c n0c-r34d 192.0.2.10 1.3.6.1.2.1.2.2.1.2

This walks ifDescr, the interface names. Use it when a single get works but the monitor still shows nothing for an interface or sensor. Interface indexes can change after a reboot or a module swap, and an agent's view can hide whole tables. On a large chassis, snmpbulkwalk is much faster.

5. tcpdump on the SNMP ports

sudo tcpdump -ni eth0 'udp port 161 or udp port 162'

This shows whether requests leave the monitor, whether replies come back, and whether traps arrive at all. Requests with no replies point at the device or a firewall. Replies the monitor then ignores point at the monitor's own settings. Save a capture with -w snmp.pcap and open it in Wireshark, which decodes SNMP so you can see the version and community string in every packet. That visibility is also the best argument for SNMPv3 on any network you do not fully trust.

Reading the error message

The wording usually names the layer. With the Net-SNMP commands:

• Timeout: No Response from 192.0.2.10 means nothing came back. Suspect the address, the community or v3 credentials (a v1 or v2c agent stays silent on a wrong community rather than refusing), the version, an ACL on the device, or a firewall.
• No Such Object available on this agent at this OID means the agent answered but does not implement that object.
• No Such Instance currently exists at this OID means the object exists but not at that index, such as an interface number that changed.
• Unknown Object Identifier comes from your own machine before any packet is sent: the MIB that defines the name is not loaded. Use the numeric OID or install the MIB.

The monitoring errors handbook covers these and many more, including graphs with gaps, counters that run backwards and email alerts that never arrive, each with checks and fixes. Always copy the exact text of an error. A paraphrase sends helpers down the wrong path.

Where to ask

Start with the documentation for the exact version you run. Most projects in the open-source software directory link their manual, community forum and issue tracker from their home page, and the Net-SNMP site has a long FAQ and tutorials for its commands. For hardware, the vendor's documentation and knowledge base tell you which MIBs a model supports and how its SNMP access lists work. General Q&A sites and forums for system and network administrators suit questions that cross several tools. Issue trackers are for bugs you can reproduce, not for "how do I" questions.

How to ask a question people can answer

The person reading your post cannot see your screen. Give them:

1. Versions: the monitoring tool, the operating system, the device model and its firmware.
2. The exact error, pasted as text, with the command that produced it.
3. What you tried and what each attempt showed. The five commands above make a good list.
4. What changed recently: an upgrade, a firewall rule, a password rotation, a replaced device.
5. Configuration, cleaned first: replace community strings, v3 passphrases, API tokens, public addresses and customer or site names with placeholders, and keep the structure.

A question built that way reads like this:

Monitoring server on Debian, polling a 48-port access switch with SNMPv2c.
Since 02:10 every interface on the switch shows "No Such Instance".
ping and mtr from the monitoring server: no loss.
snmpget of sysUpTime works and shows about three hours, so the switch rebooted.
snmpwalk of ifDescr returns 52 rows, but the indexes differ from what the monitor stored.
Nothing else changed. Should I rediscover the device, or turn on ifIndex persistence?

Anyone who has seen that problem can answer it in one line.

Frequently asked questions

My graphs have gaps, but the device never went down. Why?

Usually the poller ran late or timed out, not the device. If one polling cycle takes longer than the interval, some devices get skipped, and a jump in the monitoring server's clock can make the graph database reject updates. Compare how long a full poll takes with the interval itself; how polling works explains what to tune.

snmpwalk works from my laptop, but the monitor says timeout.

The device's ACL or a firewall allows your laptop's subnet but not the monitoring server's address, or the monitor uses a different version, community or port. Run the same command from the monitoring server and the difference shows up quickly.

Traps never show up.

Check in order: the device sends to the right address; something listens on UDP 162 (ss -ulnp will show it); the firewall allows it; the trap receiver accepts that community (in snmptrapd.conf, an authCommunity log,execute,net line); the handler runs. tcpdump tells you whether the packet arrives at all. Traps travel over UDP and can be lost, so keep polling the same objects. The SNMP traps guide has the full setup.

My traffic graph shows an impossible spike.

A 32-bit counter such as ifInOctets wrapped between polls, or the counter reset when the device rebooted, and the monitor read it as a huge increase. At 1 Gbit/s a 32-bit octet counter wraps in about 34 seconds. Poll the 64-bit ifHCInOctets with SNMPv2c or v3 instead.

The device answers ping but shows as down.

The check that failed is probably something else: an SNMP poll, a TCP port or an HTTP request. Open the host and see which check changed state, then test that one by hand.

Is it safe to paste my configuration into a forum?

Only after you remove the secrets listed above. If you posted a community string or passphrase by mistake, treat it as public and change it on every device that uses it.

Can you look at my setup?

No. We write guides and do not offer support, remote sessions or consulting. The forums and documentation above are the right place, and the five commands on this page are what anyone there will ask you to run first.