From MRTG to Prometheus: a short history of open-source network management

Open almost any traffic graph on almost any monitoring system and you will find a five-minute step in it. No standards body chose that number. It came from a Perl script written for one overloaded university link in the mid-1990s, and it has survived thirty years of rewrites.

Most of open-source network management is like that. Ideas that were good enough early got carried forward by people who forked them, rewrote them and renamed them. Knowing where those ideas came from makes today's tools easier to understand, and easier to choose between. This is a short history, with dates only where I could pin them down.

A protocol before the tools

The story starts with SNMP. The first specification appeared in 1988, and RFC 1157, published in May 1990, defined the SNMPv1 that most of us first met. In March 1991 MIB-II (RFC 1213) standardized the objects every device should expose, including the interfaces table with its octet counters and the system group with sysUpTime.

Those counters set the shape of everything that followed. A counter only goes up, and eventually wraps back to zero. It tells you nothing on its own. Somebody has to read it twice, subtract, and divide by the time in between. Every graphing tool since has been doing that sum; interface counters explained walks through it with real numbers.

SNMPv2c (RFC 1901, 1996) brought 64-bit counters and bulk requests. SNMPv3 got its lasting standards, RFC 3411 to 3418, in December 2002, finally adding real authentication and encryption. The differences between v1, v2c and v3 still matter on networks today.

For most of the early 1990s, though, the tools that used SNMP were big commercial platforms on expensive Unix workstations. A small site with one router and a Linux box had the protocol and very little to point at it.

One link, one Perl script

In 1994, De Montfort University in Leicester had a single 64 kbit/s Internet link for more than a thousand networked computers. Tobias Oetiker, who worked there, wrote a script that read the gateway router's octet counters every five minutes and drew four graphs on a web page: the last day, week, month and year. Published in the spring of 1995 as MRTG, the Multi Router Traffic Grapher, it spread quickly.

It also hit limits quickly. The first version rewrote a text log every five minutes and called outside programs for SNMP and for drawing, so one workstation could handle about twenty router ports. Dave Rand wrote rateup, a C program that took over the logging and graphing, and SNMP moved to a pure-Perl module. MRTG 2.0 shipped in January 1997.

The most important idea in MRTG 2 was easy to miss. Old data was kept at lower and lower resolution, so the log never grew. You lost detail from last year and kept the shape. That trade is still the default in half the tools you can install today.

RRDtool: the storage idea that stuck

Work on a third MRTG turned into a separate program instead. RRDtool 1.0 was released in July 1999. A round robin database is a file of fixed size, created once, holding archives at several resolutions with consolidation functions such as AVERAGE, MIN, MAX and LAST. The COUNTER data source type does the subtraction and the division for you, and copes with a counter that wraps.

The file never grows, and nobody has to run a database server. The price is that resolution is fixed when the file is created, which is why changing an interval on an RRD-based system is still a migration rather than a setting. The reasoning behind the old default is covered in choosing a polling interval.

RRDtool became the storage layer for a whole generation: Cacti, Smokeping, Munin, Ganglia, Observium, LibreNMS and many smaller projects. That is why so many network graphs from the 2000s look like close relatives.

Red dots and plugins

Graphs answer "how much". A second line of tools answered "is it up".

Big Brother, released in November 1996, showed a web page with a grid of hosts and tests, each cell a green or red dot. Its clients pushed status to the server over TCP port 1984 instead of waiting to be polled. The source was available, but under its own "Better Than Free" license rather than an open-source one. An open-source rewrite began as bbgen, became Hobbit, and was renamed Xymon to avoid trademark trouble. It is still around.

In March 1999 Ethan Galstad released NetSaint 0.0.1, a scheduler that ran small plugin programs and read their exit codes. After a trademark challenge it was renamed in 2002 to Nagios, a recursive acronym for "Nagios Ain't Gonna Insist On Sainthood". For the next decade Nagios was close to the default answer to "how do we monitor this". The plugin contract (0 for OK, 1 for WARNING, 2 for CRITICAL, 3 for UNKNOWN) was simple enough that thousands of checks were written for it, and an ecosystem grew around the core: remote agents, graphing add-ons, maps, configuration generators.

Success brought friction. In May 2009 a group of contributors, unhappy with the pace of development and how patches were handled, forked Nagios as Icinga. Checkmk started life as an add-on that fed Nagios with agent data and grew into its own distribution. The plugin format outlived all of the arguments.

The PHP years

Around the turn of the century a recipe settled in: a cheap Linux server, Apache, MySQL, PHP, cron and RRDtool. With those parts one person could build a web-based monitoring system, and many did.

Cacti had its first public release in 2001: a PHP front end that turned RRDtool's power into templates for data sources, graphs and hosts. It later gained a poller written in C for speed, and it is still maintained.

Zabbix began in 1998 as an internal tool, went public under the GPL in 2001 and reached 1.0 in 2004, with a company formed around it in 2005. Its design (a C server and agents, a PHP front end, everything in an SQL database) is recognizable in Zabbix today. OpenNMS, started in July 1999, took a different route in Java and aimed at large enterprise networks, a reminder that not everything of the era was PHP.

JFFNMS was a typical example of the PHP-era ambition. A GPL network management system written in PHP, it combined SNMP polling of routers, switches and servers with SNMP traps, syslog and TACACS+ integration, RRD graphs, autodiscovery, and a layer that consolidated raw input into events and alarms. It was packaged for Debian, Ubuntu and the FreeBSD ports tree for a while, and the project page is still on SourceForge. Tools like it tried to replace a shelf of separate scripts with one web application: poller, trap receiver, log reader and status map in a single place.

The weak spots of that generation were shared too. Pollers driven by cron could overlap when one run took longer than the interval. Busy MySQL tables became the bottleneck on larger networks. And language upgrades were unkind: PHP 7, released in December 2015, removed the original MySQL extension that a lot of older code had been written against.

Discovery, and a fork

Observium took autodiscovery further than most. Give it a device and SNMP credentials and it found the interfaces, sensors, VLANs and neighbors on its own, behind an interface people liked. In 2012 Observium moved its code away from the GPL. In 2013 a group of its users and contributors forked the last GPL-licensed code as LibreNMS, licensed it under GPLv3, and ran it as a community project. It has grown far beyond its starting point since.

By then the pattern was hard to miss: Big Brother to Xymon, Nagios to Icinga, Observium to LibreNMS. Licenses and governance decided the future of these tools as often as features did.

Metrics as a stream

Meanwhile, web companies had a different problem. They were not watching 500 router ports. They had thousands of servers and application metrics that came and went with every deploy. Graphite, written at Orbitz in 2006 and released as open source in 2008, accepted metrics pushed to it and drew graphs on demand.

Prometheus started at SoundCloud in 2012. It pulled metrics over HTTP, stored every series with a name and a set of labels, and offered a query language, PromQL, for graphs and alert rules alike. It was announced publicly in January 2015, accepted by the Cloud Native Computing Foundation in May 2016 as its second project after Kubernetes, reached 1.0 in July 2016 and graduated in August 2018. Google's SRE book later named it among the open-source systems very similar to Borgmon, Google's internal time-series alerting system.

Grafana arrived in January 2014, its interface originally based on Kibana 3. It became the common dashboard layer for Graphite, Prometheus and many other backends, including SNMP-era tools through plugins. In 2021 it moved from the Apache 2.0 license to AGPLv3.

Network gear followed through exporters, snmp_exporter among them. Look closely at a PromQL traffic query and you will find rate(): the same subtract-and-divide MRTG did against a router in Leicester.

The timeline

What carried over, and what to do with an old system

Five ideas run through the whole history: the counter and the rate, the five-minute default, the plugin exit code, templates that turn one device into fifty checks, and the fork as the community's answer to a license or governance change. When you compare tools today, as in the open-source monitor comparison, you are mostly choosing which of those ideas to put first.

If you have inherited one of the older systems, still polling away on a box in a closet, retire it gently:

1. Export the device list and the SNMP credentials before anything else.
2. Check the RRD files with rrdtool info and keep a copy with rrdtool dump if the history matters to anyone.
3. Write down every event rule and syslog pattern it uses; those are the hardest things to rebuild. The syslog setup for small networks is a good home for the log side.
4. Run the old and new systems side by side for a month and compare what each one catches.
5. Archive the old machine's disk before you power it off for good.

The software will change again. The counters, and the subtraction, will not.