Grafana dashboards that answer one question each

The first network dashboard I built had forty-one panels. Every switch uplink, every server's CPU, the UPS, the wireless controller, two weather gauges for the server room. It looked impressive on the wall and it was useless at 08:05 on a school morning, when the help desk phone starts ringing and somebody needs to know one thing: is the internet working or not. Finding that answer meant scanning forty-one tiny graphs and guessing which one mattered.

The fix was not a better layout. It was a different rule: every dashboard answers one question, and the question is its title. This piece builds one such dashboard step by step, on a Prometheus setup with blackbox_exporter, snmp_exporter and node_exporter, then shows a few more in less detail.

The setup assumed here

Nothing exotic. Prometheus scrapes three exporters:

• blackbox_exporter, running ICMP probes to the gateway, the ISP's first hop and one outside address, plus a DNS probe and an HTTPS probe.
• snmp_exporter, polling the edge router with the standard IF-MIB module, so interface metrics carry ifName and ifAlias labels.
• node_exporter on the Linux servers.

Grafana reads Prometheus as a data source. If you have not chosen your tools yet, the open-source tools comparison covers how these pieces fit next to Zabbix or LibreNMS.

Step 1: write the question down

Before opening Grafana, write the question in the exact words the person asking would use. Not "WAN overview". This:

Is the internet working for the office right now?

That sentence becomes the dashboard title. Put a second sentence in the dashboard description saying who asks it and when: "Help desk, first thing in the morning and whenever several people call at once." That description is what stops the next person from bolting six unrelated panels onto it.

# Affiliate links: we may earn a commission. We also earn from qualifying purchases on eBay.

Step 2: build the answer panel

The answer goes top left, large, and readable from across the room. For this question it is a row of Stat panels, one per hop, in path order from left to right: gateway, ISP first hop, outside address, then DNS and HTTPS. The first red box from the left is where the problem starts.

The query for each box is the share of probes that succeeded over the last five minutes:

avg_over_time(probe_success{job="blackbox_icmp", instance="10.10.0.1"}[5m])

Repeat it for 198.51.100.1 (the ISP hop) and 203.0.113.10 (the outside target), and use the DNS and HTTPS jobs for the last two boxes. In the panel options:

1. Set the unit to percent (0.0 to 1.0), so 1 reads as 100%.
2. Thresholds: red at the base, amber from 0.9, green at 1. A box that is amber means some probes failed: a flapping link, not a dead one.
3. Set the panel title to the hop name in plain words, "ISP first hop", not the IP address. The address goes in the panel description.
4. Under the standard options, set the "No value" text to "No data: check the exporter". An empty box that says nothing is worse than a red one.

Two settings at the dashboard level matter as much as the panels. Set the default time range to match the question: "right now" means the last hour, not the last seven days, where a twenty-minute outage shrinks to a sliver nobody notices. Turn on auto-refresh at about the scrape interval, a minute in my case, so a screen left open on the help desk wall is never stale.

And resist template variables on an answer dashboard. A dropdown at the top means the viewer has to make a choice before they get an answer, and the person who calls at 08:05 does not know which router to pick. Variables belong on the detailed dashboards an engineer opens next, where choosing an interface or a host is the whole point.

Five boxes, one row, and the question is answered in about two seconds. Everything else on the dashboard is evidence.

Step 3: add evidence, and stop early

Below the answer row, add only panels that help someone act on the answer. For this question, three are enough.

Round-trip time to each hop, as a time series:

probe_icmp_duration_seconds{job="blackbox_icmp", phase="rtt"}

WAN utilization against the contracted rate, not the port speed. The router's ifHighSpeed will say 1000 because the handoff is a gigabit port, but if the contract is 300 Mbit/s, that is the ceiling that matters, so divide by it:

100 * 8 * rate(ifHCInOctets{instance="edge-rtr-01", ifName="Gi0/0/0"}[$__rate_interval]) / 300e6

Add the matching ifHCOutOctets query on the same panel. $__rate_interval is a Grafana variable that picks a rate window wide enough for your scrape interval, which saves you from graphs that go blank when you zoom out. The arithmetic behind octet counters is in the interface counters article.

Total time of the DNS probe, which for a lookup is close enough to resolver response time:

probe_duration_seconds{job="blackbox_dns"}

Then stop. CPU of the core switch is interesting, but it does not answer "is the internet working", so it lives on another dashboard. My working limit is about eight panels. When I want a ninth, it usually means a second question has crept in.

Step 4: make thresholds agree with alerts

If your alert for WAN saturation fires at 90 percent for ten minutes, the utilization panel turns red at 90 percent too. If the latency alert fires at 80 ms, the RTT panel's threshold line sits at 80 ms. A dashboard that shows green while the pager is going off teaches people to distrust both. When an alert changes, change the panel in the same commit.

Step 5: link to the next question

Every answer leads to a next question. If the internet is fine but people still complain, the next question is "Is it one building or all of them?" Add a dashboard link at the top that goes there, with the time range carried across. On the WAN panel, add a data link to the full interface dashboard for that router, so the engineer who wants detail is one click away and the help desk never has to see it.

This is where the forty-one panels from the old dashboard went. Most of them still exist, spread across a handful of dashboards that each answer something, chained together with links.

Step 6: keep dashboards in files, under git

Dashboards built by clicking drift. Someone tweaks a threshold during an incident, nobody remembers why, and six months later two copies disagree. Grafana can load dashboards from JSON files at startup, which lets you keep them in a git repository like any other config.

A provider file tells Grafana where to look:

# /etc/grafana/provisioning/dashboards/network.yaml
apiVersion: 1
providers:
  - name: network
    folder: Network
    type: file
    disableDeletion: true
    options:
      path: /var/lib/grafana/dashboards/network

Export each dashboard's JSON model from its settings, save it into that directory, and commit. One snag catches everyone: the exported JSON refers to the data source by its UID, and if Grafana generated that UID, it will differ on the next server. Provision the data source with a fixed UID and the problem disappears:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    uid: prom-main
    url: http://127.0.0.1:9090
    access: proxy
    isDefault: true

Restart Grafana after adding the files:

sudo systemctl restart grafana-server

From then on, a change to a dashboard is a pull request with a diff, and rebuilding the monitoring server takes the dashboards with it.

Three more, in short

The same pattern works for most questions people bring to the monitoring screen. The answer panel and its query are the whole design; the evidence follows from them.

Each of these returns rows only when the answer is "yes, here they are". An empty table means "no", so set the "No value" text to say so in words: "No file system is on course to fill within 7 days." Otherwise an empty panel looks like a broken one.

Testing a dashboard

The test I use takes ten seconds. Open the dashboard in front of someone who did not build it, read the title aloud, and ask them to answer it. If they can, it works. If they start reading panel titles out loud and hunting, it does not, and the fix is almost always removing panels rather than adding them. The screenshots page has more on what a monitoring screen should show at a glance.

Next step

Keep the old everything-dashboard if your engineers like it, and rename it "Workbench" so nobody mistakes it for an answer. Then write down the question you were asked most often this week, word for word, and build that dashboard first. One answer row, three evidence panels, thresholds that match your alerts, and a JSON file in git.