Setting up a self-hosted monitoring server

Whichever monitoring software you settle on (Zabbix, LibreNMS, Icinga, Checkmk, Cacti, a Prometheus stack, or an older PHP and SNMP tool), the server under it needs the same few things: enough CPU for the pollers, enough disk performance for history, a clock that is right, and clear network paths to every device it watches. This chapter covers that groundwork. Use your chosen project's install guide for exact package names and commands; use this page for the decisions around them.

Sizing: what "it depends" actually depends on

Everybody asks how big the server should be, and the true answer is that it depends. It depends on four numbers you can work out before you buy or allocate anything.

The web interface matters too, but a handful of staff browsing dashboards costs far less than the polling. A wall display that redraws 40 graphs every 30 seconds is the exception.

A worked example

Take a school district with six buildings: 140 access switches (48 ports plus 4 uplinks each), 2 core switches, 4 firewalls, 40 servers, and 320 wireless access points reported through a controller.

At a 5-minute interval that is 56,100 / 300, or about 187 values per second. At a 1-minute interval it is about 935. Neither number is large for modern hardware, which is the first useful conclusion: polling a network this size is not hard work, as long as the requests run in parallel and use SNMP GETBULK (covered in the polling chapter).

Storage depends on the back end. With RRDtool, each data source keeps fixed-size archives, so the disk use is set on day one and never grows. A common layout looks like this, with a 300-second step:

RRA:AVERAGE:0.5:1:576      5-minute values for 2 days
RRA:AVERAGE:0.5:6:672      30-minute averages for 14 days
RRA:AVERAGE:0.5:24:744     2-hour averages for 62 days
RRA:AVERAGE:0.5:288:730    daily averages for 2 years
(the same four again with MAX)

That is 2,722 rows per consolidation function, 5,444 with MAX added, at 8 bytes each: about 43 KB per data source. For 56,100 data sources the total comes to roughly 2.4 GB, and it stays there. The cost of RRD files is not space but small random writes: thousands of files touched every cycle. Put them on SSD, and if the tool supports it, run rrdcached (part of RRDtool) so updates are batched.

A time-series database keeps every sample at full resolution unless you configure downsampling, so disk grows with retention. Prometheus documents its compressed storage at roughly 1 to 2 bytes per sample; planning at 1.5 bytes gives:

Tools that keep history in SQL tables (Zabbix is the common example) put that load on the database instead, and the database becomes the part of the system to size and tune first.

For this district, a virtual machine with 4 vCPUs, 8 GB of RAM and 100 GB of SSD-backed disk is a reasonable place to start. Treat it as a starting point, not a promise: measure poller duration and disk wait in the first month and adjust from evidence.

Operating system

Use a long-term-support Linux release that your team already knows: Debian stable, Ubuntu LTS, or a RHEL-compatible distribution such as AlmaLinux or Rocky Linux. Most open-source monitoring projects document one or two of these first, and the path of least resistance is the one their install guide uses. Several tools also run on FreeBSD. Avoid short-support desktop releases; you do not want an operating system upgrade forced on the monitor every nine months.

A virtual machine is fine. Two cautions: give it SSD-backed storage, and think about what happens when the hypervisor or storage it runs on fails. A monitor that dies along with everything it watches cannot tell you anything. Either host it somewhere separate or add an outside heartbeat, as described in the chapter on running a monitoring server.

The usual stack

The command-line collectors are worth installing even if your tool bundles its own, because you will use them to test devices by hand. On Debian and Ubuntu:

sudo apt install snmp snmptrapd fping rrdtool

On AlmaLinux and Rocky Linux (fping comes from the EPEL repository):

sudo dnf install net-snmp-utils net-snmp rrdtool fping

If your tool runs its jobs from cron, check that the jobs run as the tool's service account, not root, and that their output goes somewhere you will see it. Files in /etc/cron.d/ carry a user field; an illustrative /etc/cron.d/nms might read:

*/5 * * * *  nms  /opt/nms/bin/poll-all  >> /var/log/nms/poller.log 2>&1
*   * * * *  nms  /opt/nms/bin/consolidate >> /var/log/nms/consolidate.log 2>&1

Time sync comes before anything else

A monitor with a wrong clock produces events in the wrong order, graphs with gaps, and RRD errors such as "illegal attempt to update using time" when a timestamp goes backwards. Install chrony (or use systemd-timesyncd), point it at reliable sources, and point your network devices at the same sources so their log timestamps line up.

sudo apt install chrony        # or: sudo dnf install chrony
chronyc tracking
chronyc sources -v
timedatectl

Keep the server in UTC and let the interface convert to local time for display. Background on NTP itself is at the NTP project.

Packages or containers

Distribution packages update with the rest of the system but can lag well behind the project. The project's own repository or installer is usually current and is what its documentation assumes. Containers, often supplied as a Compose file, make a trial install and a clean rebuild easy. If you go that way, check three things: the database and graph data live on persistent volumes; traps and syslog reach the container with the original source address intact (many people run those receivers with host networking for that reason); and rootless containers cannot bind ports below 1024, such as 162 and 514, without extra settings.

For a first install, use whichever method the project documents as its main one.

The first hour after install

1. Change every default password: the web admin account, the database root account, any sample users. Put the web interface behind HTTPS.

2. Confirm which account the pollers and receivers run as, and that it can write to the log and graph directories.

3. Give the monitor SNMP access to devices. Prefer SNMPv3 with authentication and privacy. Where v2c is unavoidable, use a read-only community that is not public and restrict it to the monitor's address. On a Linux host running net-snmp's agent, the v2c version is one line in snmpd.conf:

   rocommunity n0c-r34d 10.20.0.50
   

4. Open the firewall paths, seen from the monitor:

   Direction	Port	Purpose
   Out to devices	UDP 161	SNMP polls
   In from devices	UDP 162	SNMP traps and informs
   In from devices	UDP 514 (TCP 6514 for syslog over TLS)	Syslog
   Out to devices	ICMP echo	Ping checks
   In from staff	TCP 443	Web interface
   Out to mail relay	TCP 25 or 587	Email actions

   # ufw: accept traps and syslog only from the management network
   sudo ufw allow from 10.20.0.0/16 to any port 162 proto udp
   sudo ufw allow from 10.20.0.0/16 to any port 514 proto udp
   
   # firewalld equivalent
   sudo firewall-cmd --permanent --add-port=162/udp --add-port=514/udp
   sudo firewall-cmd --reload
   

5. Test one device by hand before adding it to the tool:

   snmpget -v3 -l authPriv -u nmsmon -a SHA -A 'auth-secret' -x AES -X 'priv-secret' \
     10.20.4.11 1.3.6.1.2.1.1.3.0
   snmpwalk -v2c -c n0c-r34d 10.20.4.11 1.3.6.1.2.1.1
   

6. Test the inbound paths. Watch with tcpdump while you send a test trap and a test log line from another machine:

   sudo tcpdump -ni any 'udp port 162 or udp port 514'
   snmptrap -v2c -c n0c-r34d 10.20.0.50 '' 1.3.6.1.6.3.1.1.5.1
   logger -n 10.20.0.50 -P 514 -d "monitor path test"
   

   The trap OID above is coldStart. Setting up the trap receiver properly is covered in the chapter on SNMP traps.

7. Schedule a nightly dump of the configuration database from day one, and copy it off the server.

If anything in steps 5 and 6 fails, the chapter on common errors walks through timeouts, missing MIBs and silent receivers.

Pre-flight checklist

• Object count, values per object, poll interval and retention written down, with the storage arithmetic.
• SSD-backed storage for graphs and the database; rrdcached or the TSDB's own tuning in place.
• Long-term-support OS, fully patched, with automatic security updates decided on.
• chrony running, chronyc tracking showing a small offset, devices using the same time sources.
• Default passwords changed, HTTPS on, service account confirmed.
• SNMPv3 (or a restricted v2c community) configured on a first batch of devices and tested by hand.
• Firewall open for UDP 161 out, UDP 162 and 514 in, ICMP, and the web interface.
• A test trap and a test syslog line seen arriving.
• Nightly configuration backup copied off the server.
• An outside heartbeat that will notice if the monitor itself goes quiet.

With those ten lines ticked, add ten devices, not a hundred, and live with them for a few days. The next step is deciding how events become alarms, which the chapter on how monitoring data flows explains.