MTUs for batman in vxlan in WireGuard

This post is about the Maximum Transmission Unit (MTU) which has to be set on an interface for best performance.

In some cases, when no data flows through your link, it can be the MTU which is to blame.

WireGuard & PPPoE

For example, I had the problem, that some sites did not work using my WireGuard VPN at my home. This is due to PPPoE connections, which do have a smaller default MTU than the normal 1500 Bytes ¹.

Instead, they are using 1492 Bytes MTU. The WireGuard Header uses 80 Bytes, resulting in a default WireGuard MTU of 1420 - when behind a PPPoE access, we do need to decrease this by 8 accordingly to 1412 (while ¹ recommends decreasing it directly to 1280).

Freifunk

In Freifunk, we are using a software stack of WireGuard, VXLAN and B.A.T.M.A.N.

• As the routers are deployed in various places, we have to assume the minimum MTU of 1492 of our link.
• The WireGuard interface should therefore not be larger than 1412
• VXLAN uses 70 Bytes Header (14 Ethernet + 8 UDP + 8 VXLAN + 40 IPv6) - so that the MTU inside the VXLAN link is 1342
• B.A.T.M.A.N header is 32 Bytes (14 Ethernet + 18 Batman) - so that the maximum MTU we have available on our B.A.T.M.A.N link is 1310

Debugging

Behind a PPPoE carrier, we did have some weird behavior. One device did decrease performance through WireGuard VPN by 50% - surely due to the CPU being inefficient when fragmenting packages. By correcting the MTU in the WireGuard interface, however, we could not fix this by setting the MTU on various interfaces - so Vodafone seems to be weird here.

• DSL 50/10 -> no difference setting MTU setting
• DSL 100/40 with COVR -> maximum 40/20 Freifunk
• DSL 100/40 with x86 (T-Bao) -> maximum 98/38 Freifunk

Gluon

Most communities using WireGuard somehow have set the MTU to 1406, which is wrong since the MTU setting is always applied to the mesh-vpn interface - which is always the batman-carrier-network - so the VXLAN interface in our case, not the WireGuard interface ².

For best WireGuard performance, we should - according to the numbers above - have an MTU of 1342 or less in VXLAN/mesh-vpn/site.conf. This would then set the WireGuard MTU to 1412 correctly.

Minimum MTU

Now going the other way to calculate the smallest possible value for the MTU used in site.conf we can go from the minimum ipv6 MTU of 1280:

We encapsulate batman-adv frames into VXLAN and send them over UDP over IPv6 link local between gateway and nodes. This is then sent via a L3 VPN over UDP/IP. The worst case (IPv6 40 vs IPv4 20 Byte) has been assumed.

Check max MTU of the network

To have a check what the best MTU size is for your network, you can use the below script, which uses a binary search to find the best (largest working) MTU.

#!/bin/bash

# Zieladresse, die überprüft werden soll
TARGET=fe80::a1:ffff:feac:115%mesh-vpn
TARGET=fe80::1%wg_mesh
TARGET=2a02:2e0:3fe:1001:302::
TARGET=8.8.8.8

# Start-MTU-Wert. Typischerweise wird 1500 verwendet, es sei denn, es gibt einen Grund zu glauben, dass das Netzwerk anders konfiguriert ist.
MTU=1500

# Der niedrigste und höchste MTU-Wert, der getestet werden soll
LOW=1200
HIGH=1500

echo "Ermitteln der maximalen MTU zum Ziel $TARGET"

# Binary Search, um die höchste funktionierende MTU zu finden
while [ $LOW -le $HIGH ]; do
  MID=$(( (LOW + HIGH) / 2 ))
  if ping -c 1 -M do -s $((MID-28)) $TARGET > /dev/null 2>&1; then
    echo "MTU von $MID funktioniert"
    LOW=$((MID + 1))
    WORKING_MTU=$MID
  else
    echo "MTU von $MID ist zu groß"
    HIGH=$((MID - 1))
  fi
done

echo "Die maximale MTU zum Ziel $TARGET ist $WORKING_MTU"

Conclusion

In Freifunk WireGuard networks, one should have an MTU value in the site.conf which is between 1312 (to make sure that all packages inside Freifunk are sent with an MTU of 1280) and 1342 (the maximum value going from a maximum MTU of 1492).

I am not 100% sure this is correct, but wrote it with my knowledge I aggregated on this topic when writing the PR to gluon.

References:

https://www.slideshare.net/AnnikaWickert/ffmuc-half-a-year-with-WireGuard

https://github.com/freifunk-gluon/community-packages/pull/6#discussion_r470978863 https://gluon.readthedocs.io/en/latest/user/mtu.html

https://keremerkan.net/posts/WireGuard-mtu-fixes/ https://gist.github.com/nitred/f16850ca48c48c79bf422e90ee5b9d95