After my recent re-install of my fileserver I decided to make use of the Netdata monitoring (https://www.netdata.cloud). It is simple and requires very little configuration which suits me perfectly at the moment. But to my surprise it started to throw warnings at me from the start. Strange as my server is just installed and doesn’t have many service nor traffic to speak of, just a bunch of disks and NFS/CIFS shares.
One that caught my eye was Interface Drops (net_drops.enp3s0) which sounded like there was something wrong with the network interface or the local network:
A quick look at ifconfig confirms that there is packet drop on the interface. Not a large amount but enough to trigger the warning in Netdata.
# ifconfig enp3s0 enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.6 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::62a4:4cff:feb1:b0d5 prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:b1:b0:d5 txqueuelen 1000 (Ethernet) RX packets 13150808 bytes 5182069895 (4.8 GiB) RX errors 0 dropped 2874 overruns 0 frame 0 TX packets 12350768 bytes 15847850867 (14.7 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
But at the same time ethtool didn’t report these packages at all.
# ethtool -S enp3s0 NIC statistics: tx_packets: 12350894 rx_packets: 13151278 tx_errors: 0 rx_errors: 0 rx_missed: 2 align_errors: 0 tx_single_collisions: 0 tx_multi_collisions: 0 unicast: 12879297 broadcast: 40489 multicast: 231492 tx_aborted: 0 tx_underrun: 0
Odd. This is on my local network and the server is not exposed to internet so the source of those packets should be local. While I do have quite a few devices on my home network none of them should as far as I know send out unknown traffic that will get dropped just like that.
Looking at the graph I could easily see that the drops where very regular. Every 30 second there was a packet dropped.
Time to look at the interface with tcpdump and see if there is any obvious offenders that appear every 30 seconds. And behold, after some fancy filtering to remove familiar, unsuspicious, traffic this line regularly came up every 30 second:
17:53:24.402973 LLDP, length 85: UniFiSwitch
Interesting. So my Ubiquity UniFi switch (a US-8) is using LLDP, Link Layer Discovery Protocol (wikipedia), to advertise its existence on the local network. This is what gets dropped regularly as my server doesn’t understand it and thus triggering the warning in Netdata.
To solve this I decided to make my server aware of LLDP by installing the lldpd package. This version of LLDP doesn’t require and specific configuration. It “just works”.
# apt-get install lldpd # systemctl enable lldpd # service lldpd start
And within just a few minutes the warning in Netdata disappeared. Good times.
Since this was on a newly installed server with not that much traffic on the interfaces it was easy to catch. Had I started up all services these packets would have made up such a low ratio that they probably wouldn’t have triggered a warning.