What you can monitor: ports, peers, sensors and services

A host on its own tells you very little. What you actually watch are the things on it: a port, a routing session, a fan, a disk, a web service, a certificate. Monitoring tools give these different names (interfaces, items, services, sensors, entities), but every one of them has the same two possible properties. It has a state (up or down, established or idle, OK or failed), a value that changes over time (bits per second, degrees, days left), or both.

Deciding which of these to collect is most of the design work. Collect too little and outages surprise you; collect everything and nobody reads the graphs. The table below is a working catalog: what each type is, where the data comes from, what is worth graphing, and what is worth an alert. The sections after it add the commands and caveats.

The catalog

The rest of this page goes through the families in that table.

Ports, SVIs and bundles

Interfaces are defined by the Interfaces Group MIB, RFC 2863. Start by walking the names and states so you know which ifIndex is which:

snmpwalk -v2c -c n0c-r34d 10.10.0.2 IF-MIB::ifName
snmpwalk -v2c -c n0c-r34d 10.10.0.2 IF-MIB::ifAlias
snmpwalk -v2c -c n0c-r34d 10.10.0.2 IF-MIB::ifOperStatus
snmpget  -v2c -c n0c-r34d 10.10.0.2 IF-MIB::ifHCInOctets.3 IF-MIB::ifHCOutOctets.3

ifOperStatus returns up (1), down (2), testing (3), unknown (4), dormant (5), notPresent (6) or lowerLayerDown (7). Alert on "down" only when ifAdminStatus is up, otherwise every port someone shut on purpose becomes an alarm. Even then, be selective. Access ports to desks go up and down all day as people dock and undock laptops. Uplinks, trunks, server ports and anything with a description in ifAlias deserve alarms; the rest deserve a graph at most.

Use the 64-bit counters (ifHC*, SNMP v2c or v3 only) for anything faster than about 100 Mbit/s. The 32-bit ones wrap too quickly for a five-minute poll, a problem explained in interface counters explained. Errors and discards are counters too: graph them as a rate and alert on a sustained rate, not on the raw total, which only ever goes up.

Port channels deserve one more check. When one member of a two-link bundle fails, the bundle stays up and nothing turns red, but its speed halves. Watching ifHighSpeed on the bundle, or the member count, catches it.

Routing neighbors

A BGP session is either established or it is not, and bgpPeerState tells you which: idle (1), connect (2), active (3), opensent (4), openconfirm (5), established (6). The standard BGP4-MIB only covers IPv4 peers and has no per-family prefix counts, so most vendors add their own MIB for those. Many shops also turn the router's own syslog messages for neighbor changes into events, which is faster than waiting for the next poll.

snmpwalk -v2c -c n0c-r34d 10.10.0.1 BGP4-MIB::bgpPeerState

OSPF neighbors finish in full (8) when adjacency is complete, but on broadcast segments two routers that are neither designated nor backup designated router stay in two-way (4) by design. Alert on a neighbor leaving the state it normally sits in, not on anything short of full.

Hardware health

Temperature, fans and power supplies are where vendors differ most. The standard ENTITY-SENSOR-MIB covers sensors on many devices; others use their own environment MIBs, and servers usually expose the same readings through IPMI on the baseboard controller. Alert on the device's own threshold or sensor state where it reports one, because the vendor knows the safe range for that chassis better than a single number in your monitor does.

UPS units speak UPS-MIB (RFC 1628) through a network card, or report over USB to Network UPS Tools on a nearby server. The values that matter are whether the unit is on battery, the estimated runtime, the battery charge and the load. Runtime is the one to alert on, because it already accounts for load and battery age. The UPS monitoring with NUT article walks through a full setup.

Services seen from outside

A TCP connect proves something is listening. An HTTP request proves the application answers. A TLS check proves the certificate is still valid, and certificate expiry is the cheapest outage to prevent:

echo | openssl s_client -connect www.example.com:443 -servername www.example.com 2>/dev/null | openssl x509 -noout -enddate
dig @192.0.2.53 www.example.com A +short
dig +short SOA example.com @ns1.example.com

The first line prints the certificate's expiry date. The second checks that a resolver returns the address you expect, and the third, repeated for each of your name servers, shows whether they all carry the same zone serial. Run outside checks from more than one place where you can; a single vantage point cannot tell "the site is down" from "the path from the monitor is down". Ping is not monitoring compares these check types in more detail.

Services seen from inside

Some things only the host can tell you. The net-snmp agent can count processes for you with proc lines in snmpd.conf:

proc  sshd
proc  chronyd  1  1

The second line means "exactly one chronyd". The agent then publishes a count and an error flag per line in prTable (UCD-SNMP-MIB), and a non-zero prErrorFlag is your alarm. On hosts with a monitoring agent, the agent's own service checks do the same job.

Web servers publish their own status pages. For Apache, enable mod_status and let only the monitoring server read it; for nginx, the stub_status module does the equivalent:

<Location "/server-status">
    SetHandler server-status
    Require ip 10.10.0.50
</Location>

location = /nginx_status {
    stub_status;
    allow 10.10.0.50;
    deny all;
}

Requesting /server-status?auto returns machine-readable lines such as Total Accesses, BusyWorkers and IdleWorkers. Busy workers pinned at the configured maximum for several polls means requests are queuing.

Database replication needs a query rather than SNMP. On a MySQL replica, SHOW REPLICA STATUS (or SHOW SLAVE STATUS on older versions) shows whether both replication threads are running and how many seconds the replica is behind. On a PostgreSQL standby, SELECT now() - pg_last_xact_replay_timestamp(); gives the replay lag, and pg_stat_replication on the primary lists connected standbys. Alert on stopped replication first; lag thresholds depend on what the replica is for.

SLA probes

Probes measure the network itself rather than a device: routers or small agents send test traffic to each other and record round-trip time, jitter and loss. Cisco IP SLA, TWAMP responders and Smokeping all work this way. Graph them continuously, because the value is in the history: a link whose jitter creeps up every weekday at 09:00 tells a story that a single test never will.

Where to start

If you are setting up a new monitor, this order covers most outages for the least effort:

1. Reachability and uptime for every host.
2. Uplink, trunk and server ports: state, traffic, errors.
3. CPU, memory and storage on servers and hypervisors, with storage as days until full.
4. UPS runtime and on-battery state.
5. Outside checks for every service people use: TCP, HTTP, certificate expiry, DNS.
6. Routing neighbors, power supplies and fans on core gear.
7. Application details (web server workers, replication, probes) once the basics are quiet.

For how often to collect each of these, see polling. For anything not in this catalog, writing a custom check shows how to add your own.