SNMP traps and informs: receiving them with snmptrapd

Most SNMP traffic is a conversation the monitoring server starts. It asks a device for a value on a schedule and stores the answer. Traps run the other way: the device decides something happened, builds a small message and sends it to whoever it was told to notify. That makes traps fast, often well under a second after a port drops, and it also makes them unreliable in ways that polling is not.

What a trap is

A trap is a single UDP datagram sent from an agent (the SNMP service on a router, switch, UPS or Linux server) to a manager, normally on UDP port 162. Polling uses UDP 161 on the agent; traps use 162 on the receiver. A firewall rule that only allows 161 lets you poll happily and silently eats every trap.

Because it is UDP and the sender expects no reply, a plain trap is fire and forget. If the packet is lost, or sent to an address where nothing listens, nobody finds out. The agent will not retry.

Every trap carries an identity (which kind of event this is) and a list of variable bindings, or varbinds: pairs of OID and value with the details. A linkDown trap carries the ifIndex of the port plus its admin and operational status. The identity says what happened; the varbinds say where.

The generic traps

SNMPv1 defined six generic traps any agent could send, plus a seventh value meaning "read the vendor's specific number instead". SNMPv2 gave each of them a notification OID under snmpTraps.

Authentication failure traps hint that someone is scanning the management network, or that an old monitoring box still uses a retired community. On devices facing an untrusted network they can also become a flood, so many teams enable them on core gear and leave them off at the edge.

Enterprise-specific traps carry the useful detail: fan and power supply failures, temperature alarms, BGP session changes, a UPS on battery. In v1 a vendor trap is an enterprise OID plus a specific-trap number. In v2 the two become one OID: the enterprise OID, then .0, then the specific number. That rule lets one handler cover an old device and a new one reporting the same fault.

Trap, v2 notification and inform

Three message types arrive on port 162, and they are not interchangeable.

Because the v2 format dropped the agent address field, the receiver trusts the packet's source address. If devices send traps from a loopback or management interface, make sure that address is the one your inventory knows.

Informs answer the lost-trap problem: the sender waits for a Response and retransmits on timeout. That costs a little memory on the device, and it only helps while some path to the receiver exists. The operations are defined in RFC 3416, and v1 to v2 conversion in RFC 3584.

v1 and v2c traps carry the community string in clear text, exactly like a poll. Outside an isolated management network, use SNMPv3 with authPriv, or at least accept traps only from known source ranges. The trade-offs are in SNMP v1, v2c and v3: what changed.

Running a receiver with snmptrapd

Most open-source monitoring systems do not listen on port 162 themselves. They rely on snmptrapd from Net-SNMP to receive and decode traps, then pick up the result through a script, a log file or a database. The setup is the same whether the next stop is Zabbix, LibreNMS, Icinga or your own script.

1. Install the daemon: snmptrapd on Debian and Ubuntu, part of net-snmp on RHEL-family systems. Install MIB files too.
2. Write /etc/snmp/snmptrapd.conf. Since release 5.3, snmptrapd drops traps it has no access rule for, logging No access configuration - dropping trap. A fresh install that "receives nothing" is very often just this.
3. Allow UDP 162 inbound on the receiver, from management ranges only.
4. Run it in the foreground first: snmptrapd -f -Lo prints every trap to standard output. Once it works, run it under systemd and log to a file or syslog.

A small, realistic configuration:

snmpTrapdAddr udp:162
authCommunity log,execute n0c-tr4p 10.20.0.0/16

createUser -e 0x8000000001020304 trapuser SHA "replace-this-auth-phrase" AES "replace-this-priv-phrase"
authUser log,execute trapuser priv

traphandle IF-MIB::linkDown /usr/local/bin/trap-to-nms linkdown
traphandle IF-MIB::linkUp   /usr/local/bin/trap-to-nms linkup
traphandle default          /usr/local/bin/trap-to-nms other

The first line listens on UDP 162 on all IPv4 addresses, which is the default anyway. authCommunity takes the processing types log, execute and net (forward to another receiver) plus an optional source network, so the community is accepted from the management range and ignored elsewhere. For v3, notice the -e on createUser: with traps the sending device is the authoritative engine, so the receiver needs a user tied to that device's engine ID. With informs the receiver is authoritative, one reason v3 informs are easier to roll out across a hundred switches.

What a handler receives

A traphandle program reads the trap on standard input, already converted to the v2 layout even when a v1 device sent it: hostname, transport address, then one varbind per line, OID first and value after the first space. The first two varbinds are always the sender's uptime (net-snmp may print it as sysUpTimeInstance) and snmpTrapOID.0.

core-sw1.example.net
UDP: [10.20.0.11]:50612->[10.20.0.5]:162
DISMAN-EVENT-MIB::sysUpTimeInstance 12:3:41:07.15
SNMPv2-MIB::snmpTrapOID.0 IF-MIB::linkDown
IF-MIB::ifIndex.10114 10114
IF-MIB::ifAdminStatus.10114 up
IF-MIB::ifOperStatus.10114 down

The handler maps the sender to a known host, turns ifIndex 10114 into an interface name, and writes an event. Keep it quick: snmptrapd waits for each handler to finish, so slow lookups stall every trap behind it. Write to a queue and return. Script conventions are in feeding your monitor from outside programs.

Making OIDs readable

Without MIBs the receiver prints numbers. With the vendor's MIB files loaded, the same trap reads as a named notification with values like down instead of 2.

• Put vendor MIBs in a directory such as /usr/local/share/snmp/mibs, add it with mibdirs +/usr/local/share/snmp/mibs in snmp.conf, and load them with mibs +ALL.
• On Debian and Ubuntu the stock /etc/snmp/snmp.conf disables MIB loading with a bare mibs : line. Comment it out, and install snmp-mibs-downloader for the standard IETF MIBs.
• Check that a name resolves before you use it in a traphandle line. snmptranslate -On IF-MIB::linkDown should print .1.3.6.1.6.3.1.1.5.3, and snmptranslate -Td IF-MIB::linkDown shows the full definition.

Loading every MIB from every vendor slows each command and invites name clashes. Load what you have hardware for.

Testing without waiting for a failure

Net-SNMP ships snmptrap and snmpinform, which send any notification from any Linux box. Send tests from inside the management range so your source rules match. The empty '' is the uptime argument; the tool fills in a default.

snmptrap -v 2c -c n0c-tr4p 10.20.0.5 '' IF-MIB::linkDown \
  IF-MIB::ifIndex.2 i 2 IF-MIB::ifAdminStatus.2 i 1 IF-MIB::ifOperStatus.2 i 2

snmpinform -v 2c -c n0c-tr4p 10.20.0.5 '' IF-MIB::linkDown \
  IF-MIB::ifIndex.2 i 2 IF-MIB::ifAdminStatus.2 i 1 IF-MIB::ifOperStatus.2 i 2

snmptrap always exits cleanly, because nothing confirms a trap. snmpinform waits for the Response and fails on timeout, which makes it the honest test: if it times out, your traps were never arriving either.

Watch the receiver while you send, with snmptrapd -f -Lo or tcpdump -n -i any udp port 162. If tcpdump sees packets and snmptrapd logs nothing, check the access rules and community. If tcpdump sees nothing, check the path: host firewall, router ACL, or a device sending to an old address. More symptoms are in common monitoring errors.

Pointing devices at the receiver

Every device needs the receiver's address, the version and credentials, and a list of notifications to send. On a Linux server running the Net-SNMP agent, that is a few lines in snmpd.conf:

trap2sink      10.20.0.5 n0c-tr4p
informsink     10.20.0.5 n0c-tr4p
authtrapenable 1

Use trap2sink for v2c traps or informsink for informs, not both; authtrapenable 1 adds authenticationFailure. Network gear has its own syntax for the same settings. On each platform, set the trap source interface to the management address and check which trap groups are on by default. Environmental, power and routing protocol notifications often need enabling separately.

Why traps alone are not enough

Treat a trap as a fast hint, not a record of truth. The weaknesses are built in:

• UDP loss. A failing link drops packets, and the trap about that link often has to cross it.
• The broken path. When a branch router's only WAN link dies, its linkDown trap has no way out. You hear nothing from exactly the site in trouble.
• Silent devices. A switch that loses power sends nothing until it comes back.
• Drift. Readdress the receiver or change a community, and traps stop with no error anywhere.
• Storms. A flapping port can produce hundreds of traps a second. Devices rate-limit traps, so the one you needed may be the one dropped.

Traps and polling together

The pattern that works: poll everything on a steady interval, and use traps to react between polls. When a linkDown trap arrives, record an event and, if your NMS can, poll that interface at once to confirm. The poller catches the dead branch and the switch with no power. Traps catch the 20-second flap a 5-minute poll never sees. Both feed the same event stream, where duplicates merge into one alarm.

A checklist for a new trap receiver:

1. UDP 162 open on the receiver, from management ranges only.
2. authCommunity or authUser lines in place, v3 wherever the network is not trusted.
3. MIBs loaded for every vendor you run, and snmptranslate resolves their names.
4. One snmpinform test from each device family, seen end to end in the NMS.
5. Polling still covers every host and interface the traps describe.
6. A monthly look at which devices have never sent a trap. Those usually point at the wrong address.