Tracking Down Blips

In a previous article, Iexplained the process for setting up Cacti, which is agreat program for graphing just about anything.One of the main things Igraph is my internet usage. And, it's great information to have, until thereis internet activity you can't explain. In my case, there was a"blip"every 20 minutes or so that would use about 4mbps of bandwidth(Figure 1). In the grand scheme of things, it wasn't a big deal, becausemy connection is 60mbps down. Still, it was driving me crazy. I don'tlike the idea of something on my network doing things on the internetwithout my knowledge. So, the hunt began.

Figure 1. That blip drove me crazy for weeks.

Most folks immediately told me to use Wireshark to analyze thedata. That's good advice, but the problem makes me want a real-timemonitoring system rather than a one-off packet search. Plus, evenwith Wireshark, you need to address the issue of capturingall the dataflowing to and from the internet. Modern switching hardware purposefullydirects traffic only to the ports on your switch where the traffic isintended. That means you can't just "sniff" the whole network withoutsome effort. So regardless of how I was going to analyze the traffic,I had to be able to see the traffic. Thankfully, there are a few waysto accomplish that.

Network hubs were very common 20 years ago. The idea with a hub isthat the network data coming in is repeated to every port on the hub,and whichever computer the packet was intended for accepts it. Everyother computer just ignored the data. This worked fine when the amount ofdata was lowand the speed of the data was slow, but as more devices were added to thenetwork,it quickly became congested. About that time, "switching" technology enteredthe picture. A switch would accept data on every port, but repeat thepackets only to the single port on which the intended device was listening. Atfirst, switches were extremely expensive, so it wasn't uncommon tosee a four-port rackmount switch that had hubs connected to each port. Itwas a way to segregate the congestion into manageable chunks. Eventually,switching technology became mainstream. Now even the $10 eight-port devicesyou can buy online are switches instead of hubs, and the idea of a hubis outdated.

Although Ethernet networks rarely use hub technology anymore, Wi-Fi doesn'thave that same luxury. In fact, the reason Wi-Fi access points cansupport only a certain number of devices before they become unusable isthat Wi-Fi functions conceptually like a hub. All the devices on a Wi-Finetwork receive all the packets and "accept" only packets destinedfor them. That's the reason it is so important to be security-consciouswhen using a public Wi-Fi network. Anyone else connected can see allyour traffic, so make sure any sensitive data is encrypted via SSL,or even better, encrypt all your traffic via VPN.

Why is all that important? Well, if you're trying to monitor an entirenetwork, switches work against you. Back in the day of hubs, everycomputer saw all the traffic on a network, which made it easy to monitorwhat was happening. (It also made it easy to sniff other people's packets,but that's another story altogether.) Thankfully, there are a few optionsfor capturing all the data so you can analyze traffic on your network.

The first thing to determine is where you want to monitor. You canmonitor only traffic flowing through a certain place, so you need to determinewhere that place is. In my case, I want to monitor internet traffic, sofrom a port standpoint, I want to see all the traffic flowing into andout of where my router plugs in to my LAN (Figure 2). There are afew ways to capture that data—let me run through the options.

Figure 2. This is the bottleneck through which all internet traffic flows.

If you look at Figure 3, you can see how this option works. All theinternet traffic flows through the hub into the router. Because hubs repeat alldata to every port on the hub, you simply plug the monitoring computerin to that same hub, and it "hears" all the network traffic to and from theinternet. There are a few problems with this method. One, it's difficultto find hubs anymore, especially those capable of 100mbps. Even ifyou can find a hub, it will be a cheap desktop type that is likely notreliable enough to handle all your data. Quite honestly, even thoughoption 1 is technically still feasible, it's a really bad idea, and Idon't recommend it.

Figure 3. A hub is technically an option, but not a good one.

This is probably the best way to monitor network traffic in a modernLAN environment. Figure 4 shows what it looks like. The only problemwith this method is that it requires a "smart" or"managed" switch. Thatusually increases the cost drastically, but it gives you other managementfeatures like VLANs. Still, if you already have an unmanaged switch,this can be a large expenditure. If you want to use this method in anenvironment already populated with unmanaged switches, perhaps considergetting a small managed switch and put it in place of the hub shown in Figure 3.

Figure 4. This is probably the best way to monitor traffic, but itrequires a managed switch, which I didn't have.

The actual process for mirroring a port works differently on differentbrands of switches. Regardless of the specific method, however, allmanaged switches should allow you to mirror the traffic from one port toanother. Then your monitoring computer can listen on that mirror port andanalyze the traffic. It's sort of like creating an internal two-port hubthat sends traffic only one way. Usually if you do this, it's wise to"sniff" with a second Ethernet card on the monitoring computer. Thatway, you're not confusing traffic to and from the monitoring computerwith the internet traffic.

Figure 5 shows what I actually did on my network. I didn't use this methodbecause it's better than option 2; rather, I used it because I didn't havea managed switch. Basically, in this setup, you need a monitoring serverwith two Ethernet cards. You create a "bridge" interface, and plug thecomputer in between the switch and the router. This does capture all thetraffic because it literally flows through the computer. Unfortunately,it means if something goes wrong with your monitoring computer, itcan take down internet access for your entire network, along with DHCPand DNS if your router hosts those for you. There are special Ethernetcards with multiple ports that "fail open", so that if the machine does,traffic still flows. If you're going to spend money on this, however,I recommend just getting a managed switch and doing a mirrored port.

Figure 5. Although possibly not the best way to do it, this is probablythe nerdiest. It's also the way I did it, because it didn't require anynew hardware.

This option works only if your router is robust enough to runbandwidth-monitoring tools. Some are, especially if you're using a full-blownserver as a router. Most can do it, but not well. Still, if you onlywant simple monitoring and your router can handle the load, it's an easyway to do it.

I went with Option 3, which meant I needed to install a network bridgeon my Linux machine. On Ubuntu, the process is really simple. It's notdifficult on other distros like CentOS, but if you're using somethingother than Ubuntu, you'll need to google the specific steps. For Ubuntu,do:

apt-get install bridge-utils

Then edit your /etc/network/interfaces file:

# The loopback network interfaceauto loiface lo inet loopback# The bridge interfaceauto br0iface br0 inet dhcp    bridge_ports eth0 eth1    bridge_stp off    bridge_fd 0    bridge_maxwait 0

You'll need to adjust your interface names if your Ethernet devicesdon't come up as eth0 and eth1, but the configuration should make sensefrom looking at my example. You either can restart networking or, even better,reboot your system to make sure it comes up properly on startup. Onceup and running, theifconfig command should look something like mine:

spowers@pooky:~$ ifconfigbr0    Link encap:Ethernet  HWaddr 00:25:90:34:d4:3a       inet addr:192.168.1.6  Bcast:192.168.1.255  Mask:255.255.255.0       inet6 addr: fe80::225:90ff:fe34:d43a/64 Scope:Link       UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1       RX packets:1820381471 errors:0 dropped:0 overruns:0 frame:0       TX packets:124514207 errors:0 dropped:0 overruns:0 carrier:0       collisions:0 txqueuelen:1000       RX bytes:1850742830285 (1.8 TB)  TX bytes:34896441989 (34.8 GB)eth0  Link encap:Ethernet  HWaddr 00:25:90:34:d4:3a       inet6 addr: fe80::225:90ff:fe34:d43a/64 Scope:Link       UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1       RX packets:1040590501 errors:0 dropped:56421 overruns:0 frame:0       TX packets:782968757 errors:0 dropped:0 overruns:0 carrier:0       collisions:0 txqueuelen:1000       RX bytes:1101548247906 (1.1 TB)  TX bytes:493789819966 (493.7 GB)       Interrupt:16 Memory:fb5e0000-fb600000eth1  Link encap:Ethernet  HWaddr 00:25:90:34:d4:3b       UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1       RX packets:1001689311 errors:0 dropped:55961 overruns:0 frame:0       TX packets:1165557388 errors:0 dropped:0 overruns:0 carrier:0       collisions:0 txqueuelen:1000       RX bytes:797026007540 (797.0 GB)  TX bytes:1134936725326 (1.1 TB)       Interrupt:17 Memory:fb6e0000-fb700000

Note the only interface with an IP address is br0. The other twointerfaces are bridged together, so traffic can freely flow throughthem. It's interesting that this is the way distributions like Untangledwork. They create a bridge device and then filter/block/redirect trafficas it passes through.

So far, I've gotten only to the point where the monitoring computercan listen to the traffic going to and from the internet. I haven'tactually installed any software to do the listening. Quite a fewdifferent packages exist for capturing traffic and analyzing it. Depending onthe type of network trends you're looking for, you might choose a differentsoftware package from me. I actually installed a few, but rely moston BandwidthD for analyzing traffic. I'll talk more about BandwidthD,but be sure to check out some others too:

• Darkstat

• Etherape

• Ntop

• Wireshark

• BandwidthD

I like BandwidthD because it shows traffic graphs for each device on mynetwork. If you remember my initial problem, I was trying to figure outwhat device on my network was downloading something every 20 minutes. Ifigured it was a game system or cell phone stuck in a failed downloadloop or something.

Installing BandwidthD (or most any of the utilities) is a simpleapt-getinstall away. The software is most likely in your distribution'ssoftware repository, and even if the version is a little outdated,it should work perfectly fine. The only thing I needed to do is edit/etc/bandwidthd/bandwidthd.conf and set the network I wanted to monitorand the interface I wanted to listen on. Otherwise, I left everythingthe default. BandwidthD installs an Apache configuration file, so you shouldbe able to access its interface at http://server.ip.address/bandwidthd/.

After it's running for a while, you should see statistics like thosein Figure 6,which shows the top 20 bandwidth users on my network. It's fun informationto see, but if you're looking for a specific traffic pattern, you'llneed to scroll down a bit to see network graphs for every device on yournetwork. Figure 7 shows the traffic to and from my Plex Media Server. Keepin mind this is only traffic going to and from the internet, but still,you can see when friends and family were watching videos from my serverover the internet. It's important to note that although the default pageof BandwidthD shows only the top 20 users, you can click on the networkaddress to see every user who accesses the internet. It's an amazingtool for figuring out what's happening on your network.

Figure 6. I love BandwidthD; it's full of so much juicy data.

Figure 7. Having a set of graphs for each device on the network isamazingly convenient. Having it done automatically is just plain amazing.

It turns out that I couldn't find anything on my network causing thosenetwork usage spikes every 20 minutes. I looked at the graphs for everysingle device on my network and compared it to the spikes on my Cactibandwidth graphs. I just couldn't find a match. Then I realized thatmy total bandwidth graph from BandwidthD should come pretty close tomatching my WAN bandwidth graph from Cacti. And, it didn't. My entirenetwork monitoring server setup seemed to be for nothing, because Icouldn't track down what was causing the blips.

I decided to troubleshoot my Cacti installation to see if there wassomething happening every 20 minutes to cause an error. It was then that Inoticed that while the WAN interface on my router had the blip every20 minutes, the LAN side of my router (which I graph, but never reallylook at because it's just an inverse of the WAN graph) didn't have theblip. It turns out that my UniFi router has a feature that runs a speed testevery 20 minutes to graph the health of the connection. I don't rememberturning that feature on, but sure enough, it was enabled. When I disabledthe periodic speedtest, my network blips stopped.

So in the end, my network monitoring setup didn't find anything, but Idon't regret setting it up. Now I can monitor traffic easily and seewhat sort of bandwidth requirements individual devices need. In fact,the only change I plan to make is to set up my server using Option2 instead of Option 3, because I recently upgraded my server rack tomanaged switching hardware. That way if my monitoring computer dies,my internet connection stays up.

Shawn is Associate Editor here atLinux Journal, and has been around Linux since the beginning. He has a passion for open source, and he loves to teach. He also drinks too much coffee, which often shows in his writing.