Satellites and distributed polling

A single monitoring server polling everything from one place is the right way to start. There is one box to back up, one clock, one database, and every alert comes from the same machine. Sooner or later, though, one of four pressures pushes part of the work out to other machines. Depending on the tool they are called satellites, proxies, remote pollers or collectors. This guide covers when the split is worth it and how to keep a distributed setup honest.

Why split the poller

Scale. A poller works to a fixed time budget: with a 5 minute interval, every check has to finish inside 300 seconds, with margin to spare. When the poll cycle starts running long (the polling guide shows how to measure it), you can add CPU, poll less often, or spread the work across more pollers. Past a certain size only the third option keeps the interval you want.

Distance. Polling a branch office across a WAN means every SNMP request crosses that link and waits for the round trip. A few hundred interfaces at a site 80 ms away are slow and fragile to poll from the center, and quick to poll from a small box in the same building.

Firewalls and NAT. To poll devices directly, the central server needs UDP 161 open to every remote network, and traps and syslog (UDP 162 and 514) have to find their way back. A satellite at the site needs one outbound connection to the center, usually encrypted. For client networks behind NAT, it is often the only design that works at all.

Resilience. When the WAN link to a site drops, a central poller sees nothing but timeouts. A satellite at the site keeps polling, keeps the data, and hands over the full picture when the link returns, so you see what happened during the outage instead of a gap.

Common architectures

Central server with satellites is the most common pattern. Satellites collect for their site or network segment; the central server stores history, evaluates alarms and sends notifications. Configuration is made centrally and pushed out.

Active and backup central servers add a second central node that takes over when the first fails. This protects the alerting path, which satellites alone do not.

Independent sites with a combined view run a complete monitoring instance at each location, alerting locally, while a central console reads status from all of them. It suits organizations where each site has its own staff.

Real tools mix these. At the time of writing, the main open-source options work roughly like this:

Zabbix, Icinga and Prometheus all document their distributed setups in detail. The vocabulary differs more than the ideas do.

What flows where

Draw the data flows before building anything. A central and satellite design moves four kinds of traffic:

1. Configuration, from center to satellite: which hosts to poll, which checks, how often, and the credentials to use.
2. Results, from satellite to center: metric values, check states and events, each stamped with the satellite's time.
3. Local inputs at the site: traps and syslog from devices go to the satellite in the same building, not across the WAN, and the satellite forwards them as events.
4. Commands, from center to satellite: run this check now, ping that address from the remote side.

Then decide where alerting lives. Alerting from the center is simpler and gives one place to tune rules, but it stops when a site is cut off, which is often exactly when that site has a problem. Some teams let satellites send a short list of local alerts directly (UPS on battery, core switch down) over a path that does not depend on the WAN.

When the link drops

Losing a satellite should produce one alert, not four hundred. Monitor every satellite's heartbeat from the center and make each remote host depend on its satellite, so that while the satellite is unreachable its hosts show as unknown rather than down. It works the same way as the parent router dependencies for hosts.

Size the local buffer for your longest realistic outage. A worked example: a branch satellite polls 400 interfaces for 4 values each, every 60 seconds. That is 1,600 values a minute, 96,000 an hour, about 2.3 million a day. If each stored value with its timestamp and overhead takes on the order of 100 bytes, a day of backlog is roughly 230 MB. A small disk holds a week easily. The real questions are whether the tool's buffer setting allows that much, and whether the central server can take in a day of backlog without falling behind on live data.

When the link returns, the center has to accept late data with the satellite's original timestamps. Check that graphs fill the gap instead of piling everything up at the moment of reconnection, and that alarms raised from replayed data are marked as old, so nobody gets paged for a port that went down and came back eight hours ago.

Clocks

A distributed setup is only as good as its clocks. Results carry the timestamp of the node that measured them, so a satellite running two minutes fast produces shifted graphs, events out of order and, with some storage back ends, updates rejected as coming from the future. Point every node at the same NTP sources, add the clock offset of each satellite as a check of its own, and alert on drift beyond a second or so.

Security between nodes

A satellite is a sensitive machine. It holds the SNMP credentials for its site, often SNMPv3 keys, and it has a trusted connection back to the center.

• Let the satellite connect out to the center on one TCP port, so nothing inbound has to open at the site.
• Encrypt and authenticate that connection with TLS and certificates on both ends, not just a shared password.
• Give each satellite only the credentials for its own site. A stolen branch box should not hold the keys to the data center.
• Harden it like any other server: patched OS, no spare services, disk encryption if it lives in a cupboard anyone can open.
• Use SNMPv3 with authentication and privacy from the satellite to local devices, and limit device ACLs to the satellite's address.

Adding a satellite: a checklist

1. Confirm the reason first by measuring poll duration or WAN latency.
2. Pick hardware for the site. A mini PC or a small VM is usually enough for a branch with a few hundred interfaces, and the installation guide covers sizing.
3. Sync its clock and add a clock offset check.
4. Set up the outbound encrypted connection and its certificates.
5. Move the site's hosts over in small batches and compare the data with what the center collected before.
6. Point the site's traps and syslog at the satellite.
7. Add the heartbeat check and the host dependencies.
8. In a maintenance window, unplug the WAN link and confirm you get one alert, a filling buffer and a clean replay afterwards.