NetFlow, sFlow and IPFIX: finding out who is using the bandwidth

The SNMP graph on the WAN port says the link sat at 285 Mbit/s from 14:00 to 14:40. It cannot say why. Interface counters are totals: bytes in, bytes out, nothing about who sent them or where they went. To answer "who is using the bandwidth" you need flow data, and on real equipment you will meet three families of it: NetFlow, IPFIX and sFlow. They overlap enough to be confused with each other and differ enough to matter when you pick a collector or read its numbers.

This piece puts the three side by side, shows a minimal export configuration, and then works through the sampling arithmetic, because that is where most wrong conclusions come from.

Two designs under the names

There are really only two ways to describe traffic, and every format picks one.

Flow caching is the NetFlow and IPFIX approach. The router keeps a table of active flows in memory. A flow is a set of packets that share a key; classic NetFlow uses seven fields for it: source and destination address, source and destination port, IP protocol, type of service and input interface. Each packet either updates an existing entry (packet count, byte count, last-seen time, TCP flags) or creates a new one. When the flow ends or times out, the router exports one record that summarizes it.

Packet sampling is the sFlow approach. The switch picks one packet in N at random, copies the start of it (often the first 128 bytes, which covers the headers), adds some context such as input and output port, the sampling rate and how many packets went past, and sends that to the collector immediately. It keeps no flow table at all. Separately, it pushes interface counters on a fixed interval, so the collector also gets something close to SNMP counter data in the same stream.

Neither design wins in the abstract. A flow cache gives exact counts when it is unsampled, but it costs memory and CPU on the device and it delays records. Sampling costs almost nothing, which is why it lives in switch hardware, but every number it produces is an estimate.

# Affiliate links: we may earn a commission. As an Amazon Associate I earn from qualifying purchases. We also earn from qualifying purchases on eBay.

The formats in one table

A few rows deserve a note.

NetFlow v5 is still the export with the widest support, and it is IPv4 only. Its records have a fixed layout with 16-bit AS number fields, so 4-byte AS numbers do not fit. If you run IPv6 or want fields beyond the classic set, you need v9 or IPFIX.

Version 9 and IPFIX are template based. The exporter first sends a template ("the records that follow contain these fields, in this order, at these lengths") and then data records that only make sense with that template in hand. UDP has no session, so exporters resend templates on a timer. The practical consequence: after a collector restart, it may show nothing, or only part of the traffic, until the next template arrives. That is expected behavior, not a broken exporter. Check the template refresh setting before you start a long troubleshooting session.

IPFIX, defined in RFC 7011, grew out of v9 and puts the value 10 in the version field of its header, which is why people sometimes call it NetFlow v10. Its field definitions come from an IANA registry of information elements, with room for vendor-specific ones. Version 9 is documented in RFC 3954, an informational RFC that describes Cisco's format rather than setting an IETF standard.

sFlow version 5 is what current switches speak. Because each sample is a raw packet header, the collector decodes whatever it understands: IPv4, IPv6, VLAN tags, even non-IP frames. That makes sFlow good at layer 2 questions a flow cache never sees.

Timeouts, or why flow graphs spike

A flow cache exports a record when one of four things happens: the flow ends (a TCP FIN or RST is seen), it has been idle longer than the inactive timeout, it has been alive longer than the active timeout, or the cache fills up. On classic Cisco IOS the defaults were 15 seconds inactive and 30 minutes active.

That active timeout is the trap. A backup job that runs for 40 minutes produces one record after 30 minutes holding 30 minutes of bytes, and a second one when it finishes. A collector that plots bytes at the record's end time draws nothing for half an hour and then a tall spike. Better collectors spread the bytes across the flow's start and end times, but you still wait half an hour to see them. Set the active timeout to 60 seconds and the problem mostly goes away, at the cost of more export records.

sFlow has no such delay. Samples leave the switch as they are taken.

A minimal export

On a Cisco router with Flexible NetFlow, a version 9 export to a collector at 192.0.2.50 looks like this:

flow exporter EXP-COLLECTOR
 destination 192.0.2.50
 source Loopback0
 transport udp 2055
 export-protocol netflow-v9
!
flow monitor MON-IPV4
 record netflow ipv4 original-input
 exporter EXP-COLLECTOR
 cache timeout active 60
!
interface GigabitEthernet0/0/1
 description LAN side
 ip flow monitor MON-IPV4 input

Two placement decisions matter more than the syntax.

First, collect on ingress, on every interface that carries traffic you care about. Input on the LAN side catches uploads; input on the WAN side catches downloads. Mixing ingress and egress monitoring on the same box is the usual cause of traffic counted twice.

Second, think about NAT. If the router translates addresses, flows measured on the outside interface show the same public address for every host behind it. Measure on the inside interface when the question is "which machine", and outside when it is "what did the provider see".

On the collector side, nfdump is a long-lived open-source choice for storing and querying v5, v9 and IPFIX. Once files are arriving, a top ten by bytes is one line:

nfdump -R /var/cache/nfdump -s srcip/bytes -n 10

pmacct and ntopng are other common collectors, and most of them accept sFlow as well. Before you blame any collector, capture a few export packets with Wireshark, which decodes all three formats, and confirm the exporter sends what you think it sends.

Flow records are a list of who talked to whom. Restrict who can query the collector, and set a retention period on purpose rather than by accident.

Worked example: how far to trust a sampled number

Back to the 285 Mbit/s afternoon. Say the link is a 300 Mbit/s internet circuit handed off on a 1 Gbit/s port, and the edge switch exports sFlow at 1 in 1,000 on that uplink. Decimal units throughout.

Ten minutes at 285 Mbit/s is 285,000,000 × 600 ÷ 8 = 21.4 billion bytes. With an average packet of about 1,200 bytes, that is roughly 17.8 million packets, so at 1 in 1,000 the collector receives about 17,800 samples for the window. It groups them by inside address:

Each estimate is the sampled byte count multiplied by the sampling rate. The question is how far off it might be.

With random 1-in-N sampling and a large N, the number of samples a host collects behaves like a binomial count, and the relative error at about 95 percent confidence comes out near 1.96 divided by the square root of the sample count:

So the big consumer is identified beyond argument, with an error band under 2 percent. The second host could really be anywhere from about 1.3 to 1.9 Mbit/s, which is fine for this question. The third row is noise: you know the host sent something, you do not know how much.

Strictly, that formula describes packet counts. Byte estimates wobble a little more when packet sizes vary widely, but the order of magnitude holds.

Two rules fall out of it. Make decisions on rows with hundreds of samples or more, and widen the time window rather than reading single-host numbers out of a one-minute slice. And if a small, specific flow matters, such as a SIP trunk's exact usage or a figure someone will bill on, sampling is the wrong tool: use an unsampled flow cache on the router, or the interface counters themselves.

Reconciling flows with SNMP

Flow totals and interface counters will not agree to the byte, and that is normal. Most NetFlow exporters count layer 3 bytes, the IP packet itself, while the octet counters include the Ethernet header and the frame check sequence, as covered in the interface counters article. Sampling adds its own error on top. A few percent apart is fine. Twenty percent apart means something is wrong: an interface without flow monitoring, flows from only one direction, or export packets being lost.

Every format carries sequence numbers in its headers, and a decent collector reports gaps. Watch that number. Export over UDP gets dropped silently by a busy collector or a congested path, and a flow tool that quietly misses one record in ten understates everything it shows you.

Keep the SNMP polling too. Flow data answers "who", interface counters answer "how much" with less ambiguity, and together they catch each other's mistakes. When you compare raw counter readings by hand, the counter rate calculator saves some arithmetic, and choosing a polling interval explains how a 5-minute average can hide a link that is full for 90 seconds at a time.

Where to start

If the edge box is a router or firewall that speaks IPFIX or v9, start there: unsampled, active timeout 60 seconds, ingress on the inside interfaces. If your switches speak sFlow, turn it on at around 1 in 1,000 for gigabit uplinks and sparser on faster ones, and leave counter sampling enabled. Point both at the same collector.

Then, before you trust the first top-ten list, check three things: the templates have arrived, the sequence gap count is zero, and the flow total for the WAN port lands within a few percent of the SNMP graph for the same hour.