Administering a monitoring server: reports, housekeeping and backups

Installing a monitoring server takes an afternoon. Keeping it trustworthy takes an hour or two every month, and most of that work is invisible until the day someone asks for last quarter's uptime figures or the database disk fills at 2 a.m. This guide covers the administrator's side of the job: availability reports that hold up to questions, housekeeping that keeps the database small, backups you have actually restored, upgrades that keep your history, and a way to find out when the monitor itself is the thing that broke. A monthly checklist closes the page.

Availability reports

The formula, and where the downtime comes from

Every availability report starts from the same arithmetic:

availability % = (period - downtime) / period * 100

The interesting part is the downtime. It should come from alarms (an outage with a start and an end on one object), not from raw events, because a port that flaps twelve times in a minute produces twelve events but one outage. If your event pipeline does not consolidate events into alarms yet, read events and alarms first.

Clip every alarm to the report window. An outage that began at 23:40 on the last day of the previous month counts only from midnight; one still open when the window closes counts up to the window's end. Some tools quietly skip alarms that cross a window boundary, which flatters the figures in exactly the months that had long outages. Test yours with a planned outage that crosses midnight and check that the report shows it.

One object, or a group of them

Per-object availability is easy. Group availability is where reports go wrong. Take one day (1,440 minutes) and three objects on the path to a branch office application:

• the branch uplink, down 08:00 to 08:30 (30 minutes)
• the branch firewall, down 08:20 to 09:00 (40 minutes)
• the application service, down 14:00 to 14:15 (15 minutes)

The average looks best and is wrong: users at the branch could not reach the application for 75 minutes, which no average of three separate figures can show. Summing double-counts the 10 minutes when the uplink and the firewall were both down. The union is the honest number for objects in series, where any one failing breaks the service.

The union is simple to compute. Sort every down and up transition by time, keep a counter of how many objects are down, start the clock when the counter leaves zero and stop it when it returns to zero:

def union_downtime(outages, start, end):
    """outages: list of (down, up) datetimes. Returns seconds with at least one object down."""
    edges = []
    for down, up in outages:
        down, up = max(down, start), min(up, end)
        if down < up:
            edges += [(down, 1), (up, -1)]
    edges.sort(key=lambda e: (e[0], -e[1]))
    total, depth, since = 0, 0, None
    for t, step in edges:
        if depth == 0 and step == 1:
            since = t
        depth += step
        if depth == 0:
            total += (t - since).total_seconds()
    return total

For objects in parallel (two WAN links, a pair of DNS servers) the logic flips: the service is down only while every member is down, so the clock runs only while the counter equals the number of members. Decide which model each report uses and print it on the report.

Maintenance and unknown time

Planned maintenance needs a written rule before anyone asks. The usual choice is to remove it from both sides of the fraction:

availability % = (period - maintenance - unplanned downtime) / (period - maintenance) * 100

If a contract counts maintenance as downtime, report both figures side by side. The uptime SLA calculator turns a percentage into allowed minutes per month, and the article on what 99.9 percent really allows covers the measurement choices in more depth.

Time when the monitor was down, or when a parent router hid a device, is unknown, not up. Show it as its own column. A report that silently counts unknown as up will one day claim 100 percent for a month in which nobody was watching.

Housekeeping

Events and alarms

Write a retention policy and let a nightly job enforce it. A common starting point is 30 to 90 days of raw events, a year or more of closed alarms (availability reports need them), and graph history as defined by your round-robin archives or time-series retention. Purge in batches so the table is not locked for minutes during a busy poll cycle. Table and column names differ between tools; this is the shape of it:

-- MySQL or MariaDB: repeat until it affects 0 rows
DELETE FROM event_log
 WHERE created_at < NOW() - INTERVAL 90 DAY
 LIMIT 10000;

-- PostgreSQL
DELETE FROM event_log WHERE created_at < now() - interval '90 days';
VACUUM (ANALYZE) event_log;

Run it from cron at a quiet hour (03:15 is a fine habit) and make sure the timestamp column is indexed, or the purge itself becomes the slow query you are trying to avoid.

Logs and graph files

Poller, trap and web logs grow forever unless something rotates them:

/var/log/monitoring/*.log {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

RRD files keep a fixed size, but their number grows with every port you add, and files for decommissioned hosts stay behind. Files that have not been updated in a month are either orphans or a poller that has stopped writing, and both deserve a look:

find /var/lib/monitoring/rrd -name '*.rrd' -mtime +30 -print

Backups

Back up four things: the configuration (hosts, thresholds, users, notification rules), the history database, the graph data, and everything you added by hand (custom check scripts, extra MIB files, cron entries, TLS certificates).

mysqldump --single-transaction --routines nms > /backup/nms-$(date +%F).sql
pg_dump -Fc -f /backup/nms-$(date +%F).dump nms
tar -C /var/lib/monitoring -czf /backup/rrd-$(date +%F).tar.gz rrd

Two RRDtool details matter. A file copied in the middle of an update can be inconsistent, so pause the poller for the minute the copy takes or snapshot the filesystem. And RRD files are tied to the CPU architecture that wrote them; moving to a different platform means rrdtool dump to XML on the old box and rrdtool restore on the new one.

A backup you have never restored is a hope. Once a quarter, restore onto a scratch VM and open a graph from last month.

Upgrades

Read the release notes for database schema changes first. Take a full backup or a VM snapshot, then rehearse the upgrade on a copy with a copy of the database if the change is large. Put the monitor into a maintenance window so the upgrade does not page the on-call engineer for every device. Afterwards, check three things: polls are completing, graphs have new points, and a test alert reaches a human. If you hit errors you have not seen before, the troubleshooting handbook lists the usual suspects.

Monitoring the monitor

A monitoring server cannot report its own death. Give it a heartbeat that something outside receives: a cron job that calls a check-in URL every five minutes on a second small machine or a hosted heartbeat service, which alerts through a different channel when the check-ins stop.

*/5 * * * * nms curl -fsS -m 10 https://hc.example.net/ping/nms01 > /dev/null

Make the heartbeat depend on real work. Sending it only after a poll cycle finishes catches a hung poller, which a bare cron job would miss.

Capacity checks

If RRD writes are the bottleneck, rrdcached batches updates and cuts random writes sharply.

A monthly checklist

1. Run last month's availability report and read it for impossible numbers (100 percent everywhere, negative durations).
2. Confirm the nightly purge ran every night and the events table stopped growing.
3. Check free disk on every volume and note the growth rate.
4. Compare poll cycle duration with the interval.
5. List RRD files not updated in 30 days; delete orphans, fix stalled pollers.
6. Review hosts added and removed; make sure decommissioned devices are gone from monitoring.
7. Confirm last night's backup exists and is a sensible size.
8. Fire a test alert through every notification channel.
9. Check that the external heartbeat is still arriving.
10. Read release notes for your tool and its OS packages; plan any upgrade into a maintenance window.

Once a quarter, add a full restore test. It is the item people skip, and the one that matters on the bad day.