Ping is not monitoring: ICMP, TCP and synthetic checks compared

Most outage reviews I have sat through contain some version of the same sentence: "Monitoring was green the whole time." It usually was. The dashboard showed a column of healthy ping checks while the shop returned errors to every customer who tried to pay. Nobody lied. The checks answered exactly the question they were asked, and the question was too small.

A ping tells you that something at an IP address answered an ICMP echo request. That is a useful fact. It is not the same fact as "the service works", and the gap between the two is where most surprise outages live.

What a ping actually proves

ICMP echo, defined back in RFC 792, is answered by the operating system's network stack. No application is involved. A web server whose process has crashed, whose disk is full, or whose database pool is exhausted will keep answering pings at a steady 0.4 ms for as long as the kernel runs. Put a load balancer in front and it gets worse: the virtual address answers for the whole pool, including the day every server behind it is down.

What ping is good at is the network path. It notices a host that lost power, a kernel that panicked, a route that vanished, a link dropping packets, and latency that doubled when traffic shifted to a backup circuit. Tools like SmokePing are built around that strength, sending bursts of probes and graphing the spread of latency and loss over time. That is valuable context. It is not a health check for an application.

Why routers make poor ping targets

There is a second problem with ping, and it trips up experienced people. Many routers and layer 3 switches forward transit traffic in hardware, but packets addressed to the router itself, echo requests included, go up to the control-plane CPU. That CPU also runs the routing protocols, so platforms protect it with rate limits and control-plane policing, and ICMP aimed at the box is usually the first thing squeezed. The same applies to the "time exceeded" messages traceroute depends on.

The result is a router that shows 180 ms and 20 percent loss to your ping while forwarding traffic through itself without a single drop. The classic misreading is a traceroute with loss at hop 5 and none at hops 6 through 10. That is hop 5 deprioritizing ICMP to itself, not a fault. Real loss carries through to the final hop. If you want path health, ping hosts on the far side of the routers, and treat pings to the routers themselves as context, never as an alert.

Firewalls add their own noise. Plenty of networks drop ICMP at the edge entirely, and some cloud load balancers never answer it, so a failing ping can be just as misleading as a passing one.

TCP connect checks

A TCP check opens a connection to a port, completes the three-way handshake and closes it. That proves more than ping: something is listening on that port, and every firewall between the probe and the service lets the port through. The failure modes are informative too. A refused connection means nothing is listening, usually a dead process. A timeout means the packets are being filtered or the host or path is gone.

nc -z -w 3 shop.example.com 443 && echo "443 open"

The blind spot is subtle. For a listening socket, the kernel completes the handshake on its own and queues the connection until the application gets around to accepting it. An application that has hung, with its listening socket still open, keeps passing TCP checks until that queue fills. And a TCP check knows nothing about TLS, HTTP status codes or what the page says.

HTTP and TLS checks

An HTTP check speaks the service's own protocol. Request a URL, then judge the status code, the response time and ideally a string in the body. Over HTTPS the same request exercises DNS, the certificate, the TLS terminator and whatever sits between them and the application.

curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' --max-time 5 https://shop.example.com/healthz

This catches the failures users actually report: 5xx responses, a page that loads in nine seconds, an expired or mismatched certificate, a redirect loop, a "down for maintenance" page that returns 200. Certificate expiry deserves its own check that looks ahead rather than waiting for the day it breaks. This exits non-zero if the certificate expires within 14 days (1,209,600 seconds):

echo | openssl s_client -connect shop.example.com:443 -servername shop.example.com 2>/dev/null \
  | openssl x509 -noout -checkend 1209600

What you fetch matters as much as how. A /healthz endpoint that returns "ok" without touching anything is a TCP check with extra steps. One that also queries the database and the cache, with short timeouts, tells you far more. Do not make it so deep that one slow minor dependency pages the whole rotation, though. And remember that an HTTP check only covers the path you request. A home page served from cache can look perfect while login and checkout are broken.

Synthetic transactions

A synthetic transaction does what a user does: load the page, log in with a test account, search, add an item to the cart and stop before payment, or complete it against the payment provider's test mode. It can run as a sequence of HTTP requests or in a headless browser, which also catches broken JavaScript.

These catch what nothing else does: a broken flow, a third-party login or payment provider having a bad day, one step that has quietly become slow. They also cost the most. They break when the interface changes, they need test accounts and data that stay out of real reports, and they are heavier to run, so every 5 to 15 minutes is typical rather than every 30 seconds. Give each synthetic an owner. An unowned synthetic fails on a cosmetic change, gets muted, and is never turned back on.

Where you check from, and how often

A check from inside your network tests your network. A check from outside tests what users see, including DNS, the CDN and the ISP paths. Use at least three external locations and alert when two of them agree, so one probe with a bad upstream does not page anyone at 3 a.m.

Then do the numbers against your target. A 99.9 percent monthly target allows 43.2 minutes of downtime in a 30-day month; 99.99 percent allows 4.32 minutes. The uptime and SLA calculator does this for any target and period. With 1-minute checks and a three-failures rule, detection alone takes about three minutes, already around 7 percent of a 99.9 percent monthly budget and roughly 70 percent of a 99.99 percent one. With 5-minute checks, a 4-minute outage can fall entirely between two checks and never appear in the report. If your target has more than three nines, you need automatic failover, because humans cannot respond inside the budget. The interval trade-offs are the same ones covered in choosing a polling interval.

Measured uptime is only as honest as the check behind it. Compute it from ping and you will report 100 percent for a month in which checkout failed for an hour. More on how those percentages are defined and measured is in the uptime math article.

A layered plan for one web service

Here is how I would cover shop.example.com: a load balancer, two application servers, one database.

1. ICMP from inside to each server and the database host, every 30 seconds. Dashboard only, no alerts. When something else fires, it tells the on-call engineer where to look first.
2. TCP to port 443 on the public address from three external locations, every 60 seconds. A ticket if one location fails for 10 minutes, since that is usually the probe's own network.
3. HTTPS GET /healthz expecting 200 and the string "ok" within 5 seconds, every 60 seconds from the same three locations. Page when two locations fail three checks in a row.
4. A deeper health endpoint that checks the database and cache, every 2 minutes from inside. A ticket on failure, a page only if it stays failed for 10 minutes.
5. Certificate expiry once a day. A ticket at 21 days, a page at 7.
6. DNS resolution from two external resolvers every 5 minutes, compared against the expected addresses.
7. A synthetic login, search and add-to-cart every 10 minutes from two locations. Page after two consecutive failures from both.
8. Host metrics (disk, memory, CPU, the database's connection count) through SNMP or an agent, with trend alerts that open tickets.

The pattern is deliberate: page on symptoms users feel (checks 3 and 7), open tickets for causes and early warnings (everything else). The SRE material published at sre.google makes the same split between symptoms and causes, and it is the single most effective cure for alert noise I know of. Every paging check should also link to a runbook; the runbooks article covers what goes in one, and the alert fatigue piece covers keeping the pages rare enough that people still read them.

Before you call a service monitored

• At least one check speaks the service's own protocol, not just ICMP.
• One check walks a real user path from end to end.
• Checks run from outside your network, from at least three locations, with a two-of-three rule.
• Certificate expiry warns at least two weeks ahead.
• Pings to routers are context on a dashboard, never a page.
• The check interval is small next to your monthly error budget.
• Every paging check has an owner and a runbook.
• Uptime reports come from the user-facing checks, not from ping.

If you can tick all eight, a green dashboard finally means something.