Organizing monitored hosts: names, groups, maps and discovery

Every monitoring system starts with a list of hosts. A host is anything with an address you can reach: a core switch, a firewall, a hypervisor, a printer, the network card in a UPS, a cloud VM. On each host you watch a set of objects (ports, services, sensors, certificates), and the catalog of things worth watching has its own chapter. This one is about the list itself: what to call things, how to group them, how to draw them, and how to keep the list honest while the network changes underneath it.

It sounds like housekeeping, and it is. It is also the part that decides whether an alert at 02:40 tells the on-call person what broke and where, or sends them to a spreadsheet first.

Names that survive a 3 a.m. page

A good host name answers four questions without a lookup: where is it, what does it do, which one of several is it, and (sometimes) which environment does it belong to. A common pattern is site, role, number:

hq-core-sw01      headquarters, core switch, first of a pair
hq-core-sw02      headquarters, core switch, second of a pair
wh1-acc-sw03      warehouse 1, access switch, third
br2-fw01          branch 2, firewall
hq-vmh04          headquarters, hypervisor host 4
hq-ups01          headquarters, UPS network management card

A few rules keep a scheme like this usable for years:

• Use lower-case letters, digits and hyphens only. That is what DNS host names allow (RFC 1123), and it lets the monitoring name, the DNS name and the device's own sysName be the same string.
• Put the site first. Sorted lists then group by location for free, and a filter on br2- catches a whole branch.
• Keep role codes short and written down. A two-line legend (acc access switch, dist distribution, fw firewall, vmh hypervisor, ap access point) beats clever abbreviations nobody else can read.
• Number with two digits from the start. sw01 to sw12 sorts correctly; sw1 to sw12 does not.
• Keep IP addresses, people's names and model numbers out of the name. Addresses change, people leave, and the replacement switch is a different model.

Set the device's own name to match. On most network gear the configured hostname is what SNMP returns as sysName.0, so a one-line check compares what the monitor believes with what the device says:

snmpget -v2c -c n0c-r34d 10.10.0.2 SNMPv2-MIB::sysName.0 SNMPv2-MIB::sysLocation.0

(n0c-r34d is a made-up read-only community. Anywhere outside a trusted management network, use SNMPv3 rather than a community string.)

Sites, zones and groups

Most tools let you hang hosts on more than one hierarchy. The words differ (zones, locations, host groups, device groups, folders, tags), but they come down to two ideas.

A site or zone is where a host physically or logically lives, and each host belongs to exactly one. Sites drive the settings that really are tied to location: which poller or proxy talks to the host when you run distributed polling, which discovery ranges apply, and which maintenance window covers a branch with its own ISP.

Groups and tags are everything else, and a host can carry many. The ones that earn their keep:

Keep the list short. Five dimensions with consistent values do more than twenty tags that half the hosts carry. Tags only help filtering, alert routing and reports when every host has them, so make the important ones mandatory at the moment a host is added.

Maps and submaps

Maps are groups you can see. Most monitoring tools offer some kind of status map: an icon per host or group, colored by the worst open alarm, with lines for links. A map of sixty icons is readable. A map of six hundred is wallpaper.

The answer is nesting. The top map shows one icon per site. Click a site and you get its submap: the core pair, the distribution layer, one icon standing in for all the access switches, the firewall and the uplink. Each icon on a parent map takes the worst status of everything inside it, so a red site still means "look here" without showing every port at once.

Draw maps in the shape people already think in. Help-desk staff think in buildings and floors; network engineers think in layers and links. If you have both audiences, build two views from the same hosts rather than one picture that suits neither.

Parent and child dependencies

When a branch firewall dies, every switch, phone and printer behind it stops answering too. Without dependencies you get thirty alarms for one fault, and the one that matters is buried in the middle. With them, the monitor knows that br2-fw01 is the parent of everything at branch 2, marks the children unreachable rather than down, and can hold back their notifications.

Every major tool has a version of this. Nagios and its descendants use a parents line on the host definition, Icinga 2 uses separate Dependency objects, Zabbix attaches dependencies to triggers, and LibreNMS lets you set parent devices. The rule is the same everywhere: point each host at the next device between the monitoring server and that host.

define host {
    use         generic-switch
    host_name   br2-acc-sw01
    address     10.30.1.11
    parents     br2-fw01
}

Two cautions. First, the tree is drawn from the monitor's point of view, not the users'. If the monitoring server sits at headquarters, the headquarters firewall is the parent of every branch firewall, even though branch staff never think about it. Second, keep the tree current. A dependency pointing at the wrong parent can quietly swallow a real alarm, which is worse than having no dependency at all. How alert rules use these relationships is covered under triggers and actions.

Auto-discovery and its pitfalls

Adding fifty switches by hand is slow and error-prone, so most tools can find hosts for you. The common methods:

1. Subnet sweeps. Ping or probe every address in a range, then try SNMP on whatever answers. With nmap, a ping-only scan of one subnet is nmap -sn 10.20.0.0/24.
2. Neighbor crawling. Ask each known device for its LLDP neighbors (LLDP-MIB) or, on Cisco gear, its CDP neighbors (cdpCacheTable in CISCO-CDP-MIB), add them, and repeat. This follows the real cabling and turns up switches nobody remembered.
3. Table harvesting. Read ARP and routing tables from routers to learn which subnets and addresses are in use.
4. Identification. For each new host, read sysObjectID.0 (1.3.6.1.2.1.1.2.0) for the vendor and model family, and sysDescr.0 for the OS string. The number after 1.3.6.1.4.1. is the vendor's private enterprise number: 9 is Cisco and 8072 is the net-snmp project, so a Linux server running the net-snmp agent answers with an OID under 1.3.6.1.4.1.8072.

snmpget -v2c -c n0c-r34d 10.20.0.1 SNMPv2-MIB::sysObjectID.0 SNMPv2-MIB::sysDescr.0

Discovery saves hours, and it also makes a mess if you let it run unattended. The usual problems:

• Duplicates. A router with five addresses gets found five times. Match on something stable (sysName, the serial number in ENTITY-MIB's entPhysicalSerialNum, or a chassis MAC address) before adding anything.
• Too much. Guest Wi-Fi clients, desk phones and every laptop on DHCP turn into hosts. Exclude client ranges, and accept only hosts that answer SNMP or match a list of allowed sysObjectID prefixes.
• Credential spraying. Trying every community string and SNMPv3 user against every address sends those secrets (in clear text, for v1 and v2c) to devices you do not control. Tie each credential set to the ranges it belongs to.
• Runaway scope. Neighbor crawling over a WAN can follow a link straight into an ISP's or a partner's network. Set a hop limit and explicit allowed ranges; a subnet calculator helps you write those ranges down exactly.
• Silent acceptance. Prefer discovery that proposes new hosts for a person to approve over discovery that adds them and starts alerting immediately.

If your main question is "what is plugged in where", a tool built around layer 2 neighbor crawling, such as Netdisco, is worth running alongside the monitor.

A worked layout for a three-site company

Picture a firm with a headquarters, a warehouse twenty minutes away on a site-to-site VPN, and a small sales branch: about 70 hosts in all.

The configuration that follows from it:

• Three sites, one per row, each with its own discovery range and with the Wi-Fi and DHCP client pools excluded.
• Role tags from the legend above, which pick templates: switches get port and uplink checks, hypervisors get CPU, memory and datastore checks, UPS hosts get battery and runtime checks.
• Criticality: firewalls, core switches and hypervisors are p1, access switches p2, printers and the door controller p3. P3 pages nobody and shows up on the morning report instead.
• A top map with three site icons and the VPN links between them, a submap per site, and a separate WAN view with only the firewalls and tunnels.
• Dependencies from every host to its site firewall, and from each remote firewall back to hq-fw01, so a dropped VPN to the warehouse produces one alarm rather than twenty.

Keeping the inventory honest

The host list starts drifting from reality the day you finish building it. A monthly pass catches most of the drift:

1. List hosts that have been unreachable for 30 days or more. Each one is either retired (remove it, and its history, on purpose) or broken (open a ticket).
2. Compare the host list with DHCP reservations, IPAM and DNS. Anything in IPAM but not monitored is a gap; anything monitored but missing from IPAM is undocumented. The IPAM side is covered in documenting your network.
3. Compare each host's monitoring name with its sysName.0. A mismatch usually means a device was replaced or renamed without anyone telling the monitor.
4. Check that every host has a site, a parent and the mandatory tags.
5. Work through the discovery proposals you have been ignoring.

Then add one line to your change process: nothing gets racked or retired unless the same ticket updates the monitor. That habit keeps the list closer to the truth than any discovery schedule.