Sep 29, 2023

How to debug networks with tcpdump, ping on Linux

Nowadays, I learned a lot about network debugging through my recent hobby of providing free internet access to everybody (in my local area) through Freifunk.

Looking at the backbone, many things can happen which are unusual in typical networks. For example, we have asymmetric routing, where packets can come in from one supernode, are routed to the client and the response is then sent through a different router.

In such situations it is needed to have a good tooling at hand.

Most tools is quite basic and already known like ip, ping, mtr, dig and tcpdump - but you can do a lot with them.

Basic debugging

One way to debug this is:

  1. start mtr to a target address and see where it ends
  2. login into the last known hop and see if your traffic arrives there using tcpdump -i INTF host SRC_HOST
  3. check which path this packet will take using something like ip r get ipv6::addr from src::addr
  4. check if packet is leaving the host on the expected interface using tcpdump -i INTF
  5. repeat for other systems until the problem source is found

If a firewall exists, one can use it to log catched/dropped packets (temporary of course). This is especially helpful if a firewall rule might be wrong, if packets are dropped which are needed.

Unknown source

If the source of our package is unknown it might be helpful to filter with tcpdump -i INTF icmp6 to see only ICMP packets like ping. Looking into the available routes on the host is often very helpful: ip -6 r or looking at the iptables configuration: ip6tables-save

In some cases rules can interplay also ip rule show.

If ping does not work with the external ip address, it might help to test with a local or link-local ip address, by looking at ip a and pinging with the found one.

To see if a neighbour is found on a given ipv6 address, one can use ndisc6:

ndisc6 ipv6::addr eth0 which answers with the found ip address of the neighbour, which should be correct too.

If the neighbour is not found, one can look into the neighbour list here and search for the ipv6 address:

ip -6 n

maybe check if it works through link local address:

ping fe80::4c09:f0ff:fe26:381%eth0 (don’t forget the %intf which is always needed when handling link-local addresses)

On a bridge configured by proxmox, it might be impossible to have a link-local connection. This can be solved by resetting the disable_ipv6 flag:

sysctl net.ipv6.conf.tap148i0.disable_ipv6=0
ndisc6 fe80::4c09:f0ff:fe26:381 tap148i0
tcpdump -i tap148i0 icmp6
ip a show tap148i0

I then checked if it at least works when I manually set the ipv6 neighbour like:

ip -6 n replace 2a00:fe0:43::148 dev eth0 lladdr 4e:09:f0:26:03:81 router REACHABLE

which did work - but did not clear up why it did not work before. As often, a reboot helped for now. But I really wish that I won’t see such network problems any time soon..