Syslog for small networks: collect, rotate and actually read your logs

Syslog is the oldest monitoring feed most of us have, and the one we read least. Every switch, router, firewall, access point and Linux box on your network is already writing down what happens to it. Left alone, a network device keeps those lines in a small buffer in RAM. The buffer is gone the moment the device reboots, which is exactly when you wanted to read it.

The fix is not glamorous: one Linux machine running rsyslog, listening on UDP 514, writing one folder per device, with logrotate keeping the disk in check and every clock on the network agreeing on the time. I run this setup for the school district's network and on a shelf at home, and the two configs differ by a handful of lines. Here is the whole build, in the order I would do it.

Two formats on the wire

Syslog was in use for years before anyone wrote it down. RFC 3164 (2001) described what BSD-style systems were already doing. RFC 5424 (2009) replaced it with a proper standard and moved transports into separate documents, with RFC 5426 covering UDP. You will receive both, often from the same rack.

The older style:

<38>Jan 27 03:14:07 web01 sshd[2211]: Accepted publickey for backup from 192.0.2.50 port 51122 ssh2

The same event in RFC 5424 form:

<38>1 2025-01-27T03:14:07.412Z web01.example.net sshd 2211 - - Accepted publickey for backup from 192.0.2.50 port 51122 ssh2

The number in angle brackets is the priority, and it packs two values: facility times 8, plus severity. Here 38 is facility 4 (auth) times 8, plus severity 6 (informational).

What the newer format adds is mostly about time and structure. The RFC 3164 timestamp has no year, no time zone and no fractions of a second. RFC 5424 uses a full timestamp with a zone and up to microsecond precision, splits out the application name, process ID and a message ID, and allows structured data in square brackets (the two hyphens above mean "none"). It also raises the size a receiver should accept: the old format topped out at 1024 bytes.

Network gear is often in between. Cisco IOS sends something close to the old format, often with a message counter in front and no hostname at all. That is one reason the server below files messages by the sender's IP address rather than by the name inside the message.

Facilities and severities

Severity is the field you will filter on. Selecting a level means that level and everything more severe, so warning gets you 0 through 4.

Facility says which part of the system spoke. The ones you will meet most:

So a Cisco interface going down arrives as priority 187 (local7 is 23, times 8 is 184, plus severity 3). Once you can do that sum in your head, raw packet captures stop looking like noise.

Step 1: build the server

Any small Linux machine will do: a VM with one vCPU and 1 GB of RAM, an old desktop, or the Pi from the weekend home-lab build. Put the log folder on its own disk or partition so a chatty device can never fill the root file system.

On Debian or Ubuntu:

sudo apt install rsyslog
sudo mkdir -p /var/log/remote
# Ubuntu only: rsyslog runs as the syslog user there and must own the folder
sudo chown syslog:adm /var/log/remote

Then create /etc/rsyslog.d/10-remote.conf:

# One file per sending device, named by source IP
template(name="PerHost" type="string"
         string="/var/log/remote/%fromhost-ip%/messages.log")

# Everything that arrives over the network goes through this rule set only
ruleset(name="remote") {
    action(type="omfile" dynaFile="PerHost"
           template="RSYSLOG_FileFormat"
           dirCreateMode="0750" fileCreateMode="0640")
}

module(load="imudp")
input(type="imudp" port="514" ruleset="remote")

Three choices in there are deliberate. Binding the input to its own rule set keeps remote messages out of the server's own log files. Naming folders by %fromhost-ip% means a device that sends no hostname, or the wrong one, still lands in a predictable place; if you have reverse DNS for your management network, %fromhost% gives you names instead. And RSYSLOG_FileFormat writes full timestamps with time zone and fractions, so lines from different devices sort correctly.

If your main rsyslog.conf already loads imudp, remove the module() line here, because loading a module twice is a config error. Want TCP as well? Add module(load="imtcp") and a matching input(type="imtcp" port="514" ruleset="remote").

Check the syntax, restart, and confirm the port is open:

sudo rsyslogd -N1
sudo systemctl restart rsyslog
sudo ss -ulnp | grep ':514'

Syslog over UDP has no authentication: anything that can reach port 514 can write lines into your logs. Allow only the management network:

sudo ufw allow from 192.168.10.0/24 to any port 514 proto udp

Test from another Linux machine, then look for a new folder named after its IP:

logger -n 192.168.10.5 -P 514 -d "syslog test from $(hostname)"
ls /var/log/remote/

The full option list for every module is in the rsyslog documentation.

Step 2: point the devices at it

Cisco IOS. The logging source-interface line matters more than it looks: it pins the source address, and our folders are named after it.

conf t
 service timestamps log datetime msec localtime show-timezone
 logging source-interface Vlan10
 logging host 192.168.10.5
 logging trap informational
end
write memory

logging trap informational sends severities 0 to 6 and leaves debug on the box, which is what you want.

Linux hosts running rsyslog. Create /etc/rsyslog.d/90-forward.conf, then run rsyslogd -N1 and restart:

*.info  action(type="omfwd" target="192.168.10.5" port="514" protocol="udp")

If a machine keeps its logs only in the systemd journal, install rsyslog first and make sure local messages appear under /var/log before you add forwarding.

OpenWrt, from an SSH session on the router:

uci set system.@system[0].log_ip='192.168.10.5'
uci set system.@system[0].log_port='514'
uci set system.@system[0].log_proto='udp'
uci commit system
/etc/init.d/log restart

Most other things on a small network (NAS boxes, access points, printers, UPS network cards) have a syslog server field somewhere in the web interface. Fill in the address, leave the port at 514, choose UDP.

Step 3: rotate before the disk fills

Create /etc/logrotate.d/remote-syslog:

/var/log/remote/*/messages.log {
    daily
    rotate 90
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    postrotate
        systemctl kill -s HUP rsyslog.service
    endscript
}

That keeps 90 days, compresses everything except yesterday's file, and sends rsyslog a HUP afterwards, which makes it close its files and open fresh ones on the next write. Dry-run it, then force one rotation to prove it works:

sudo logrotate -d /etc/logrotate.d/remote-syslog
sudo logrotate -f /etc/logrotate.d/remote-syslog

If logrotate refuses with a complaint about insecure permissions on the parent directory, add a su line inside the block naming the user and group that own the folders.

For sizing, do the arithmetic for your own network. Forty devices sending 2,000 lines a day at about 150 bytes a line is 12 MB a day, or roughly 1 GB over 90 days before compression, and log text compresses very well. The real risk is one device left in debug mode. Once a month, run du -sh /var/log/remote/* | sort -h and look at the bottom line.

Step 4: agree on the time

Logs from five devices are only useful together if their clocks agree. Point everything at the same NTP source and keep devices in UTC, so nobody has to do time-zone sums during an outage.

On the server, timedatectl should say the clock is synchronized, and chronyc tracking shows the offset if you run chrony. On Cisco IOS:

conf t
 clock timezone UTC 0
 ntp server 192.168.10.1
end
show ntp associations

OpenWrt runs an NTP client out of the box; point it at the same server in LuCI under System, then the Time Synchronization tab. If one device still has the wrong time, rsyslog keeps the time it received each line as a separate property, timegenerated, which you can add to a template while you chase the clock.

Reading what you collected

Plain tools go a long way:

# Everything one switch said today
less /var/log/remote/192.168.10.21/messages.log

# Which devices had ports flapping, and how often
grep -c 'LINEPROTO-5-UPDOWN' /var/log/remote/*/messages.log

# Who changed a config, across current and rotated files
zgrep -h 'SYS-5-CONFIG_I' /var/log/remote/*/messages.log*

# Watch every device live while you work on one
tail -F /var/log/remote/*/messages.log

On the Linux hosts themselves, the journal is often quicker than the forwarded copy:

journalctl -p err -b
journalctl -u ssh.service --since "2025-01-27 02:00" --until "2025-01-27 04:00"
journalctl -t CRON --since yesterday
journalctl -f

The first shows errors and anything worse since boot. The unit is sshd.service on the Red Hat family.

Reading is good; being told is better. A few patterns deserve to raise an event in your monitoring system: a config change outside a maintenance window, a power supply failure, a port flapping more than a few times an hour. The field manual covers turning syslog lines into events with regular expressions. Keep the list short, or you will meet the problem described in alert fatigue.

When files stop being enough

Flat files and grep carry a small network a long way. You have outgrown them when several people search the logs every day, when you need answers across months in seconds, when you want counts and graphs by field, or when an auditor asks who can read what. That is the point to add a log store such as Graylog, Grafana Loki or OpenSearch. Some monitoring tools also take syslog directly, LibreNMS among them; the open-source monitor comparison covers where each one fits.

Keep the rsyslog server either way. It stays the collector every device points at, and one extra line inside the remote rule set sends a copy onward:

action(type="omfwd" target="192.168.10.6" port="1514" protocol="tcp")

Use whatever port your log store listens on. If the new system has a bad week, the flat files are still there.

The checklist

• One server, with /var/log/remote on its own disk or partition.
• UDP 514 open to the management network only.
• One folder per device, named by source IP, with a fixed source interface on routers and switches.
• Severity informational and above from network gear, never debug.
• logrotate daily, compressed, 90 days, tested with -d before you trust it.
• NTP on every device, everything in UTC.
• A saved grep for each event you care about, and a monthly look at which device talks most.