Repository files navigation

This application is a NetFlow/IPFIX/sFlow collector in Go.

It gathers network information (IP, interfaces, routers) from different flow protocols,serializes it in a protobuf format and sends the messages to Kafka using Sarama's library.

The diversity of devices and the amount of network samples at Cloudflare required its own pipeline.We focused on building tools that could be easily monitored and maintained.The main goal is to have full visibility of a network while allowing other teams to develop on it.

In order to enable load-balancing and optimizations, the GoFlow library has adecoder which convertsthe payload of a flow packet into a Go structure.

Theproducer functions (one per protocol) then converts those structures into a protobuf (pb/flow.pb)which contains the fields a network engineer is interested in.The flow packets usually contains multiples samplesThis acts as an abstraction of a sample.

Thetransport provides different way of processing the protobuf. Either sending it via Kafka orprint it on the console.

Finally,utils provide functions that are directly used by the CLI utils.GoFlow is a wrapper of all the functions and chains thems into producing bytes into Kafka.There is also one CLI tool per protocol.

You can build your own collector using this base and replace parts:

• Use different transport (eg: RabbitMQ instead of Kafka)
• Convert to another format (eg: Cap'n Proto, Avro, instead of protobuf)
• Decode different samples (eg: not only IP networks, add MPLS)
• Different metrics system (eg: useexpvar instead of Prometheus)

The sampling protocols can be very different:

sFlow is a stateless protocol which sends the full header of a packet with router information(interfaces, destination AS) whileNetFlow/IPFIX rely on templates that contain fields (eg: source IPv6).

The sampling rate in NetFlow/IPFIX is provided byOption Data Sets. This is why it can take a few minutesfor the packets to be decoded until all the templates are received (Option Template andData Template).

Both of these protocols bundle multiple samples (Data Set in NetFlow/IPFIX andFlow Sample in sFlow)in one packet.

The advantages of using an abstract network flow format, such as protobuf, is it enables summing over theprotocols (eg: per ASN or per port, rather than per (ASN, router) and (port, router)).

Collection:

• NetFlow v5
• IPFIX/NetFlow v9
  • Handles sampling rate provided by the Option Data Set
• sFlow v5: RAW, IPv4, IPv6, Ethernet samples, Gateway data, router data, switch data

Production:

• Convert to protobuf
• Sends to Kafka producer
• Prints to the console

Monitoring:

• Prometheus metrics
• Time to decode
• Samples rates
• Payload information
• NetFlow Templates

Download the latest release and just run the following command:

./goflow -h

Enable or disable a protocol using-nf=false or-sflow=false.Define the port and addresses of the protocols using-nf.addr,-nf.port for NetFlow and-sflow.addr,-slow.port for sFlow.

Set the brokers or the Kafka brokers SRV record using:-kafka.brokers 127.0.0.1:9092,[::1]:9092 or-kafka.srv.Disable Kafka sending-kafka=false.You can hash the protobuf by key when you send it to Kafka.

You can collect NetFlow/IPFIX, NetFlow v5 and sFlow using the same collectoror use the single-protocol collectors.

You can define the number of workers per protocol using-workers .

We also provide a all-in-one Docker container. To run it in debug mode without sending into Kafka:

$ sudo docker run --net=host -ti cloudflare/goflow:latest -kafka=false

To get an example of pipeline, check outflow-pipeline

The samples flowing into Kafka areprocessed and special fields are inserted using other databases:

• User plan
• Country
• ASN and BGP information

The extended protobuf has the same base of the one in this repo. Thecompatibility with other softwareis preserved when adding new fields (thus the fields will be lost if re-serialized).

Once the updated flows are back into Kafka, they areconsumed bydatabase inserters (Clickhouse, Amazon Redshift, Google BigTable...)to allow for static analysis. Other teams access the network data just like any other log (SQL query).

If you want to develop applications, buildpb/flow.proto into the language you want:

Example in Go:

PROTOCPATH=$HOME/go/bin/ make proto

Example in Java:

export SRC_DIR="path/to/goflow-pb"export DST_DIR="path/to/java/app/src/main/java"protoc -I=$SRC_DIR --java_out=$DST_DIR $SRC_DIR/flow.proto

The fields are listed in the following table.

You can find information on how they are populated from the original source:

• ForsFlow
• ForNetFlow v5
• ForNetFlow v9
• ForIPFIX

If you are implementing flow processors to add more data to the protobuf,we suggest you use field IDs ≥ 1000.

The pipeline at Cloudflare is connecting collectors with flow processorsthat will add more information: with IP address, add country, ASN, etc.

For aggregation, we are using Materialized tables in Clickhouse.Dictionaries help correlating flows with country and ASNs.A few collectors can treat hundred of thousands of samples.

We also experimented successfully flow aggregation with Flink using aKeyed Session Window:this sums theBytes x SamplingRate andPackets x SamplingRate received during a 5 minuteswindow while allowing 2 more minutesin the case where some flows were delayed before closing thesession.

The BGP information provided by routers can be unreliable (if the router does not have a BGP full-table or it is a static route).You can use Maxmindprefix to ASN in order to solve this issue.

Licensed under the BSD 3 License.