Netdata warns about packets dropped ratio

After my recent re-install of my fileserver I decided to make use of the Netdata monitoring (https://www.netdata.cloud). It is simple and requires very little configuration which suits me perfectly at the moment. But to my surprise it started to throw warnings at me from the start. Strange as my server is just installed and doesn’t have many service nor traffic to speak of, just a bunch of disks and NFS/CIFS shares.

One that caught my eye was Interface Drops (net_drops.enp3s0) which sounded like there was something wrong with the network interface or the local network:

A quick look at ifconfig confirms that there is packet drop on the interface. Not a large amount but enough to trigger the warning in Netdata.

# ifconfig enp3s0
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.6  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::62a4:4cff:feb1:b0d5  prefixlen 64  scopeid 0x20<link>
        ether 60:a4:4c:b1:b0:d5  txqueuelen 1000  (Ethernet)
        RX packets 13150808  bytes 5182069895 (4.8 GiB)
        RX errors 0  dropped 2874  overruns 0  frame 0
        TX packets 12350768  bytes 15847850867 (14.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

But at the same time ethtool didn’t report these packages at all.

# ethtool -S enp3s0
NIC statistics:
     tx_packets: 12350894
     rx_packets: 13151278
     tx_errors: 0
     rx_errors: 0
     rx_missed: 2
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 12879297
     broadcast: 40489
     multicast: 231492
     tx_aborted: 0
     tx_underrun: 0

Odd. This is on my local network and the server is not exposed to internet so the source of those packets should be local. While I do have quite a few devices on my home network none of them should as far as I know send out unknown traffic that will get dropped just like that.

Looking at the graph I could easily see that the drops where very regular. Every 30 second there was a packet dropped.

Time to look at the interface with tcpdump and see if there is any obvious offenders that appear every 30 seconds. And behold, after some fancy filtering to remove familiar, unsuspicious, traffic this line regularly came up every 30 second:

17:53:24.402973 LLDP, length 85: UniFiSwitch

Interesting. So my Ubiquity UniFi switch (a US-8) is using LLDP, Link Layer Discovery Protocol (wikipedia), to advertise its existence on the local network. This is what gets dropped regularly as my server doesn’t understand it and thus triggering the warning in Netdata.

To solve this I decided to make my server aware of LLDP by installing the lldpd package. This version of LLDP doesn’t require and specific configuration. It “just works”.

  # apt-get install lldpd
  # systemctl enable lldpd
  # service lldpd start

And within just a few minutes the warning in Netdata disappeared. Good times.

Since this was on a newly installed server with not that much traffic on the interfaces it was easy to catch. Had I started up all services these packets would have made up such a low ratio that they probably wouldn’t have triggered a warning.