Skip to content

Polling Interval

Uptime & Alerting, Tech Notes

Runbooks an on-call engineer can use at 3 a.m.

Ellen Hartigan8 min read

When I led a help desk, the knowledge-base articles that mattered were never the long ones. They were the short pages a new hire could follow on their first evening shift, alone, with a queue building up. On-call runbooks are the same thing with higher stakes. The reader is capable, but tired, interrupted, and possibly looking at this alert for the first time in their life.

This piece gives you a template, some rules for writing to that reader, and a complete example you can copy and adapt.

What a runbook is, and what it is not

A runbook is a short, specific procedure tied to one alert or one symptom. It says what the alert means, how bad it is, what to check first, what is safe to do, who to call, and how to undo anything you did.

A runbook is not:

  • A design document. The reasons the network was built this way belong somewhere else, linked if useful.
  • A vendor manual. Nobody needs the full command reference at 3 a.m., only the four commands that answer this question.
  • A postmortem. Lessons from past incidents should change the runbook, but the story of the incident lives in its own report.
  • A script with no explanation. Automation is welcome, but the reader still needs to know what it does and when not to run it.
  • A substitute for fixing the alert. If the honest runbook says "ignore this, it clears by itself", the alert should be deleted or demoted. Alert fatigue covers how to make that call.

The template

Every runbook on a team should have the same shape, so the reader always knows where to look. This is the one I use:

RUNBOOK: <alert name, exactly as it appears in the notification>
Owner: <team>    Last reviewed: <date>    Last used: <date>
Alert rule: <link to the rule in the monitoring system>

1. What this alert means
2. Impact
3. First checks (read-only)
4. Safe actions
5. Escalation
6. Rollback

1. What this alert means

One or two sentences in plain words. Name the device, the port or service, the condition, and how long it has to last before the alert fires. If the reader understands only this section, they should still know what broke.

2. Impact

Who is affected, how badly, and how to tell "degraded" from "down". This section also answers the question every tired person asks first: can this wait until morning? Say so directly.

3. First checks

Read-only steps, ordered from most likely cause to least likely. Each step gives the exact command and what good and bad output look like. Nothing in this section should change anything.

4. Safe actions

Things that cannot make the situation worse, or that are quick to reverse, each with the condition under which to take it. Anything with user impact is marked clearly. A short "do not" list belongs here too.

5. Escalation

Roles, not personal names, because names go stale faster than anything else in a document. Say when to escalate (a time limit, or a change in impact) and what to bring with you when you do.

6. Rollback

For every action in section 4, the way to undo it. If the reader changed any configuration, how to get back to the saved version.

Writing for a tired reader

The person reading your runbook has less patience and less short-term memory than you do while writing it. A few habits help a lot:

  • One action per numbered step. If a step contains "and then", split it.
  • Commands that can be copied and pasted as written, with real hostnames, real interface names and real indexes. Placeholders are where mistakes happen.
  • Show what the output should look like. "Look for line protocol is up" is far better than "check the interface".
  • Write decisions as jumps: "If the port says err-disabled, go to step 4b."
  • Put the time limit near the top. "If you are not making progress after 20 minutes, escalate" gives people permission to ask for help.
  • Avoid "simply", "just" and "obviously". Nothing is obvious at 3 a.m.
  • Spell out abbreviations that only one person on the team uses.
  • If it scrolls past two screens on a phone, split it into two runbooks.

The best test I know is the help-desk test: hand the runbook to someone who did not write it, on a quiet afternoon, and watch where they hesitate. Every hesitation is an edit.

A runbook that has to be searched for will not be found. Put its address inside the notification itself, so it is one tap away on a phone. Most tools have a field for this. Nagios has notes_url on hosts and services, Zabbix triggers have a URL field, and Prometheus rules commonly carry a runbook_url annotation:

- alert: CoreUplinkDown
  expr: ifOperStatus{device="core-sw-01", ifName="Te1/1/1"} == 2
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "core-sw-01 Te1/1/1 (to dist-b-01) is down"
    runbook_url: "https://wiki.example.com/runbooks/core-uplink-down"

Repeat the link in the message body as well, because some paging apps and SMS gateways drop custom fields. The field manual page on triggers and actions shows where the message template lives in a typical setup.

Keeping runbooks current

A runbook that was right last year can be dangerous this year. A few small rules keep them honest:

  • Every runbook has an owner and a "last reviewed" date, and the owning team reviews its runbooks every quarter.
  • Whoever uses a runbook during an incident updates the "last used" date and fixes one thing that slowed them down. Small fixes, often.
  • Every change ticket that touches a device asks one question: "Does a runbook mention this device?" If yes, the runbook is updated before the ticket is closed.
  • Postmortems end with a runbook action item, even if it is "no change needed".
  • Once a quarter, during a quiet hour, someone new to the alert follows the runbook against a lab device or a planned maintenance window.
  • When an alert is deleted, its runbook is deleted too.

It also helps to keep runbooks next to the console people already use. If the operators' first-15-minutes routine (see working the monitoring console) starts on the alarm list, the runbook link should be on the alarm.

Here is a complete runbook for a small campus. Building B has one distribution switch, dist-b-01, with two 10-gigabit uplinks: one to core-sw-01 and one to core-sw-02. Commands use Cisco IOS syntax, so adjust them for your platform.

RUNBOOK: Uplink down: core-sw-01 Te1/1/1 (to dist-b-01)
Owner: Network team    Last reviewed: 2026-01-12    Last used: 2025-11-03
Alert rule: https://nms.example.com/rules/core-uplink-down

1. What this alert means

The 10-gigabit link from core-sw-01 port Te1/1/1 to the Building B distribution switch dist-b-01 has been down for at least 2 minutes, and nobody shut it down on purpose (the port is still administratively up).

2. Impact

If the second uplink (core-sw-02 Te1/1/1) is up, users notice nothing, but Building B has no redundancy. This can wait until business hours once you have confirmed step 3a. If the second uplink is also down, Building B is offline and a separate alert, "Building B unreachable", will fire. Follow that runbook instead.

3. First checks (read-only)

a. Confirm the second uplink is up. This decides everything else.

core-sw-02# show interfaces TenGigabitEthernet1/1/1 | include line protocol
TenGigabitEthernet1/1/1 is up, line protocol is up (connected)

b. Look at the failed port.

core-sw-01# show interfaces TenGigabitEthernet1/1/1 | include line protocol

"down, line protocol is down (notconnect)" means no light or no link: go to 3c. "(err-disabled)" means the switch shut the port itself: go to 4b.

c. Check the optics. Very low or missing receive power points at the fiber, the patch, or the far end.

core-sw-01# show interfaces TenGigabitEthernet1/1/1 transceiver detail

d. Find when it happened and whether it flapped first.

core-sw-01# show logging | include 1/1/1

Several up and down messages before the final drop usually mean a dirty or damaged fiber. If you can, open the error graph for this port in the NMS. Rising CRC errors in the hours before the drop support the same conclusion (see interface counters explained).

e. If you cannot log in to the switch, check from the monitoring server with SNMPv3. The ifIndex for Te1/1/1 on core-sw-01 is 49.

snmpget -v3 -l authPriv -u nms-ro -a SHA -A "$SNMP_AUTH" -x AES -X "$SNMP_PRIV" \
  10.20.99.2 IF-MIB::ifAdminStatus.49 IF-MIB::ifOperStatus.49

Expected while the alert is active: ifAdminStatus is up(1) and ifOperStatus is down(2). The command comes from Net-SNMP.

f. Check the change calendar for work in Building B or on either core switch in the last 24 hours.

4. Safe actions

a. Second uplink up and port in notconnect: change nothing. Acknowledge the alert with a note ("B redundancy lost, physical fault likely"), open a ticket for a fiber and optic check during business hours, and stop here.

b. Port err-disabled and second uplink up: bounce the port once.

core-sw-01# configure terminal
core-sw-01(config)# interface TenGigabitEthernet1/1/1
core-sw-01(config-if)# shutdown
core-sw-01(config-if)# no shutdown
core-sw-01(config-if)# end

Wait 2 minutes and repeat check 3b. Do not save the configuration.

c. Do not reload either core switch, change spanning-tree settings, or ask anyone on site to move cables at night. None of those are safe without a second person.

5. Escalation

  • Network on-call secondary: if both uplinks are down, if the bounce in 4b did not bring the port up, if the port flaps, or after 20 minutes without progress.
  • Facilities on-call: if the Building B closet has lost power or someone needs to inspect the fiber patch panel.
  • Bring the output of 3b, 3c and 3d, and the time the alert fired.

6. Rollback

  • After 4b the port should be administratively up, as it was before. Confirm with show running-config interface TenGigabitEthernet1/1/1 and compare with the last config backup.
  • If the port comes up and starts flapping (more than two changes in 5 minutes), shut it to stop the churn, escalate, and write "Te1/1/1 left shut by on-call" in the ticket so the day shift re-enables it after the repair.

That is the whole runbook. It fits on a phone screen or two, it answers "can this wait?" in section 2, and it tells the reader when to stop. If you write only one thing this quarter, write this runbook for the alert that wakes your team most often, then hand it to someone who has never seen that alert.

More on this layer