Don’t use your Ubiquiti USG for speed tests

No, you can not trust the built-in speed test in the Ubiquiti Unifi Security Gateway or the controller. Here are some rather simple test that I made.

First of all, my internet connectivity is probably above average. I have two ISPs where the primary is a 1000/1000 connection (WAN1 on the USG) and the secondary is a 250/50 connection (WAN2 on the USG).

Everything in my network is connected by cable. Incoming internet is delivered to me with either standard gigabit ethernet cable (cat-6) or via a coax connected cable modem. All computers and devices at home are connected by cable too except mobile phones and tablets. These tests except the controller test is made using my iMac connected like:

iMac -> Unifi Flex Mini -> Unifi US-8 -> Unifi Security Gateway -> Internet

This is the built in speed test in my Unifi Controller (version 6.0.45.0) running on a Raspberry Pi 3B:

So yes, just under 200/200 isn’t that impressive on a supposed 1000/1000 connection. But I know I have downloaded files much, much faster than that. Something is not quite right here. Let try a test from my computer. As I’m Swedish there is a local initiative called Bredbandskollen (http://www.bredbandskollen.se/) which is the go-to test that most ISPs recommend here. Lets see what I get there:

Wow, 900Mbps and less than a millisecond in latency! That’s more like it. And this also proves that the USG very well can handle speeds up to gigabit without much problems (provided the correct settings, I have “Protection Mode” set to “Disabled” as afaik it doesn’t do much anyway other than provide a false sense of security).

But one test is no test (well, now we have two tests that shows different results) so I did a few more. Note that these tests were done while my home network was resonable un-quiet with many browser tabs open, Spotify playing music etc.

FAST (http://fast.com) is the one that is recommended by Netflix I think. Gives very similar results to Bredbandskollen.

And this is SpeedTest by Ookla (http://speedtest.net) which gives even slightly higher speeds.

Conclusion: Don’t ever trust the built-in speed test that the USG and/or controller provides (unless you have a sub-100 Mbps connection I guess). Always (and I can not stress this enough) do the tests with a wired connection. Going wireless introduces so many variables that is hard to control. Always do at least two (three) tests using different services. These tests are of course only showing the speed I get at one point in time and if I suspected variance I would have to do more tests of a longer time period.

Bonus information for me: I really get the promised 1000/1000 speeds that my primary ISP is selling me.

Netdata warns about packets dropped ratio

After my recent re-install of my fileserver I decided to make use of the Netdata monitoring (https://www.netdata.cloud). It is simple and requires very little configuration which suits me perfectly at the moment. But to my surprise it started to throw warnings at me from the start. Strange as my server is just installed and doesn’t have many service nor traffic to speak of, just a bunch of disks and NFS/CIFS shares.

One that caught my eye was Interface Drops (net_drops.enp3s0) which sounded like there was something wrong with the network interface or the local network:

screenshot from Netdatas notification list

A quick look at ifconfig confirms that there is packet drop on the interface. Not a large amount but enough to trigger the warning in Netdata.

# ifconfig enp3s0
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.6  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::62a4:4cff:feb1:b0d5  prefixlen 64  scopeid 0x20<link>
        ether 60:a4:4c:b1:b0:d5  txqueuelen 1000  (Ethernet)
        RX packets 13150808  bytes 5182069895 (4.8 GiB)
        RX errors 0  dropped 2874  overruns 0  frame 0
        TX packets 12350768  bytes 15847850867 (14.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

But at the same time ethtool didn’t report these packages at all.

# ethtool -S enp3s0
NIC statistics:
     tx_packets: 12350894
     rx_packets: 13151278
     tx_errors: 0
     rx_errors: 0
     rx_missed: 2
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 12879297
     broadcast: 40489
     multicast: 231492
     tx_aborted: 0
     tx_underrun: 0

Odd. This is on my local network and the server is not exposed to internet so the source of those packets should be local. While I do have quite a few devices on my home network none of them should as far as I know send out unknown traffic that will get dropped just like that.

Looking at the graph I could easily see that the drops where very regular. Every 30 second there was a packet dropped.

screenshot of the graph Interface Drops

Time to look at the interface with tcpdump and see if there is any obvious offenders that appear every 30 seconds. And behold, after some fancy filtering to remove familiar, unsuspicious, traffic this line regularly came up every 30 second:

17:53:24.402973 LLDP, length 85: UniFiSwitch

Interesting. So my Ubiquity UniFi switch (a US-8) is using LLDP, Link Layer Discovery Protocol (wikipedia), to advertise its existence on the local network. This is what gets dropped regularly as my server doesn’t understand it and thus triggering the warning in Netdata.

To solve this I decided to make my server aware of LLDP by installing the lldpd package. This version of LLDP doesn’t require and specific configuration. It “just works”.

  # apt-get install lldpd
  # systemctl enable lldpd
  # service lldpd start

And within just a few minutes the warning in Netdata disappeared. Good times.

Since this was on a newly installed server with not that much traffic on the interfaces it was easy to catch. Had I started up all services these packets would have made up such a low ratio that they probably wouldn’t have triggered a warning.