May 7, 2024

BGP and OSPF explaination tutorial

Disclaimer: This post is intended for a technical interested audiance with experience in routing between computer systems, especially linux.

You might, or might not want to read the first part of this guide, though this longer post explains a little bit more about the basics of BGP.

For Freifunk Aachen, we created a BGP setup with OSPF as an internal protocol. This allowed me to grasp the configuration of BGP and get more into advanced networking stuff.

More specifically, our use-case is as follows:

We have two backbone VMs (bb-a and bb-b), which speaks BGP through bird v1 towards Freifunk Rheinland (through 6 ipv4 gre tunnels which have a BGP session for ipv4 and ipv6 each).

For simplicity reasons, we can just talk about a single backbone VM here called bb-a.

Inside of our RZ, we want to talk OSPF in our network and create a VM which has a FFRL IPv4 and IPv6 which is announced through OSPF to bb-a, making it able to communicate with these IP addresses into the internet.

For simplicity reasons, we are creating this example for IPv4 only - with the target of announcing 185.66.193.46/32 to bb-a.

Our VM which should be able to communicate is called VM-157 and is reachable through the public ipv6 address 2a00:fe0:43::157. We want to give

Basic concepts - technical introduction

First of all, lets take a look at a few of the protocols used in this tutorial:

Proxmox

Proxmox is the hypervisor, which manages the VMs (virtual machines) and has quite advanced networking capabilities through its SDN (software defined network) config. Equivalent tools are Oracle VM VirtualBox, VMware ESXI or Microsoft Hyper-V - proxmox generally makes it much easier to work with VMs than it would be the case when using the utilized virtualization technology QEMU directly.

Bird

Bird is a Internet routing daemon which can speak OSPF, RIP and BGP protocols. In general bird manages importing and exporting routing information between different protocols. Typically, the protocols are kernel, OSPF, static and BGP in this scenario. One can configure this neatly by selecting parts from one of the protocols using import filters, which are added to the bird internal table. And the other way makes it possible to export statements from the bird table into one of the protocols using filters as well.

This can be imagined as following.

graph LR; $protocol-->import_statement; import_statement-->bird; bird-->export_statement; export_statement-->$protocol;

If the single global bird table is not enough, one can also add a second table (instead of the default master table). To transfer between internal bird tables one can then use the protocol pipe which is more of an advanced feature.

Just like all good tools, bird has a weird version break from v1 to v2 which changed the syntax a little, while making it possible to declare ipv4 configs and ipv6 configs in the same file instead of having a separate daemon per ip version. But we will see more of that later on.

So far so good. We still need to get a basic understanding of the other two routing protocols.

BGP

The Border Gateway Protocol makes it possible to announce routes towards other machines (so called neighbors) through a peering. One can configure a peering with another node through something like this in your bird.conf:

router id 100.64.5.237;
define AS_FFAC = 65077;
define AS_FFRL = 201701;

protocol bgp name_of_peering {
	description "My Description";
	local as AS_FFAC;
	import all;
	export all;
	neighbor 100.64.5.236 as AS_FFRL;
	source address 100.64.5.237;
};

Which would create a peering between our node and the neighbor node, while the neighbor node is declared as a different AS. We would import all the routes which are announced from that neighbor into our bird table and also export everything from our bird table into BGP - announcing it to our neighbors.

For more information it makes sense to first read a bit about Autonomous systems (AS)

OSPF

The Open-Shortest-Path-First (OSPF) protocol is a link-state-protocol - which means, that it retrieves its information by checking the state of the interface link and tries to see nodes on the other side of the link and find the best routing towards them.

Internally it maintains a link-state table which is updated through dijkstra algorithm dynamically. When every node publishes the same information to all other nodes, this leads to all nodes having the same information and the exact same routing table and weighs in the link state table.

A very good explaination of the internals of OSPF was found in this great talk from one of the people of Freifunk Rheinland: https://media.ccc.de/v/routingdays16-6-dynamisches_routing_mit_ospf

It is rather simple to configure the link but also gets its power from the import and export possibilities of bird.

Outline

To accomplish our routing announcement, we have the following steps as a small outline, which are:

  1. Establish OSPF communication
  2. Configure the IP address on VM-157
  3. Announce the 185.66.193.46/32 address through OSPF to bb-a
  4. Configure bb-a to hand this announcement from OSPF over to FFRL through BGP
  5. Correctly configure the routing on bb-a so that this node also has a route to 185.66.193.46/32 in its kernel table

We will start at the top and move our way through to a working configuration.

1. Establish OSPF communication

We need to add a internal interface to bb-a and VM-157 through the proxmox UI and add an IP address in our internal network accordingly - as OSPF still needs to have an ip address on the interface, even though it is a link-state-protocol.

We are using 100.64.17.157/24 for VM-157 and 100.64.17.141/24 for bb-a.

Now we need to configure the /etc/bird/bird.conf at the older bb-a node with the old bird syntax:

function is_default() {
	return net ~ [ 0.0.0.0/0 ];
};

protocol ospf IGP {
	import all;
	export where is_default(); # announce default route via ospf
	
	area 0 {
		interface "lo" {
			stub yes;
		};
		interface "ens20" {
			cost 10;
			hello 5; retransmit 2; wait 10; dead 20;
			check link;
			authentication none;
		};
	};
};

As the VM-157 does not have a default route, we later want to announce exactly that, as it does not need to know anything else but the default route. This is done by our export statement, which increases safety to not send wrong configurations to the other node. On the other hand we are importing everything we get from OSPF into bird - but we could also limit this to only 185.66.193.46/32 related routes.

Now lets take a look at OSPF with bird v2 on the VM-157:

protocol ospf v2 IGP {
 	ipv4 {
		import all;
		export all;
	};
	area 0 {
		interface "eth1" {
			cost 10;
			hello 5; retransmit 2; wait 10; dead 20;
			check link;
			authentication none;
		};
	};
}

Currently, we are not restricting anything and are importing everything we can get. We could of course add an import where is_default() on this side, but as we do have both nodes under our control, we don’t have to.

Finally, we can run birdc configure check as well as birdc configure if the former succeeded.

Now we should be able to see the OSPF topology using birdc show ospf topology:

> birdc show ospf topology
BIRD 2.0.12 ready.

area 0.0.0.0

	router 5.145.135.141
		distance 10
		router 5.145.135.142 metric 100
		network 100.64.17.0/24 metric 10

	router 100.64.17.157
		distance 0
		network 100.64.17.0/24 metric 10

	network 100.64.17.0/24
		dr 5.145.135.141
		distance 10
		router 5.145.135.141
		router 100.64.17.157

Here we can see our own announcement as well as the dr (direct route) to 5.145.135.141 - which is bb-a.

2. Configure the IP address on VM-157

On the VM-157 node we need to configure the 185.66.193.46/32 address on itself. We can do so using systemd-networkd to add it to the local lo interface.

[Match] 
Name=lo

[Address]
Address=185.66.193.46/32

Then run sudo service systemd-networkd reload

3. Announce the 185.66.193.46/32 address through OSPF to bb-a

We also need to import the new address into OSPF by defining the device protocol on VM-157:

log syslog all;
router id 100.64.17.157;
debug protocols all;

protocol device {
	scan time 8;
}

protocol direct {
	interface "lo";
	ipv4;			# Connect to default IPv4 table
	ipv6;			# ... and to default IPv6 table
}

The above imports our local available ip addresses into the bird table (even if there is no import stated here, this is somehow not needed with protocol direct), which is then exported because auf the export all; to OSPF. Therefore, the bb-a can now see our announced route to our address 185.66.193.46/32.

We can run birdc configure check as well as birdc configure if the former succeeded.

We can see that using birdc show route on VM-157 which show something like:

BIRD 2.0.12 ready.
Table master4:
185.66.193.46/32     unicast [direct1 2024-05-04] * (240)
	dev lo

on bb-a birdc show route shows:

BIRD 1.6.8 ready.
100.64.17.0/24     dev ens20 [direct1 2024-05-04] * (240)
                   dev ens20 [IGP 2024-05-04] I (150/10) [5.145.135.141]
185.66.193.46/32   via 100.64.17.157 on ens20 [IGP 2024-05-04] * E2 (150/10/10000) [100.64.17.157]

So we have a route to 185.66.193.46/32 from bb-a in the bird table.

4. Configure bb-a to hand this announcement from OSPF over to FFRL through BGP

On bb-a we already have the announcement in our bird table, so we only need to add the following config to BGP to forward that to FFRL as well. We have a template ffrl_exit, which is used by all the 6 BGP-Sessions to FFRL, so we only need to edit the following once:

filter hostroute {
        if net ~ 185.66.193.46/32 then accept;
        reject;
};

function is_our_prefixes() {
	return net ~ [
		185.66.193.40/29+
	];
};


template bgp ffrl_exit {
        local as AS_FFAC;
		import where is_default(); # security feature to let FFRL only announce a default route
        export filter hostroute;
        export where is_our_prefixes(); # only announce what belongs to us

        next hop self;
        multihop 64;
};

This configuration chains two filters together and also shows how function and filter both can be used in the same config and only has slight syntax differences.

One of the two would be enough here.

We are importing only the default route from FFRL into our bird table, which then exported through OSPF to VM-157.

We are declaring ourselves as the next hop of the addresses we are announcing, as FFRL will not know anything about VM-157.

After entering birdc configure check as well as birdc configure if the former succeeded, we should now see the default route on VM-157.

> birdc show route
BIRD 2.0.12 ready.
Table master4:
0.0.0.0/0            unicast [IGP 2024-05-04] * E2 (150/10/10000) [5.145.135.141]
	via 100.64.17.141 on eth1

But internet does not work yet, as we did not export the route from ospf into our kernel table on VM-157 yet. This is done in the next step.

5. Correctly configure the routing on VM-157 so that it uses the correct source IP

Just adding a kernel protocol with export all; does not help here, as we need to communicate to bb-a through our local 185.66.193.46 address to receive a response - as the internal 100.64.17.157 is not available outside bb-a (and we are not going to use NAT, that’s why we are doing real routing in the first place - right?). We need to add the following as protocol kernel above of our OSPF configuration

protocol kernel {
	ipv4 {			
		# Connect protocol to IPv4 table by channel
		export filter {
			if net ~ 100.64.17.0/24 then reject;

			krt_prefsrc=185.66.193.46;
			accept; 
		};
	};
}

# Another instance for IPv6, skipping default options
protocol kernel {
	ipv6 { export all; };
}

Looking at our config, we are setting two things in our export configuration:

  1. Add 185.66.193.46 as a preferred source to the kernel routing table (krt_prefsrc)
  2. but limit that to not overwrite the config for 100.64.17.0/24 as we still need to communicate through the 100.64.17.157 source here - otherwise our OSPF session is killed

Note:

An alternative to setting the preferred source here would be to add a SNAT iptables rule like this: iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 185.66.193.46 ! -d 224.0.0.6 which routes everything on eth1 through 185.66.193.46 except for the OSPF multicast traffic.

While we can declare ipv4 and ipv6 in the same configuration using bird2 - we still need to have separate protocol sections most of the time. This is not the case for BGP - so the BGP configuration indeed gets a lot smaller using bird2.

So now we can ping any IP-Adress but still do not receive a response. Inspecting the packages with tcpdump -i eth1 icmp while having a ping running shows, that they are sent to bb-a though.

So we need to configure something there as well..

5. Correctly configure the routing on bb-a so that this node also has a route to 185.66.193.46/32 in its kernel table

Keeping the ping running on VM-157 and running tcpdump -i ens20 icmp on bb-a shows, that there are packages received which are not able to be routed.

This is the case as we did not forward the traffic from

Historically, bb-a already has a kernel protocol config:

protocol kernel{
#       learn;          # Learn all alien routes from the kernel
#       persist;        # Don't remove routes on bird shutdown
        metric 64;      # Use explicit kernel route metric to avoid collisions
        export all;     # export all routes from bird into kernel routing table
        kernel table 42; # Kernel table to synchronize with (default: main (0))
};

Which exports all the announcements to table 42, which can be seen using ip r s t 42 which is short for ip route show table 42.

However we need to send our traffic from 185.66.193.46 into table 42 - at this point I am not that clear why this is the case.

We therefore need to add an ip rule to forward the traffic into table 42 - we can do this using ip rule add from 185.66.193.40/29 lookup 42 or using systemd-networkd:

[Match]
Name=ens20

[Address]
Address=100.64.17.141/24

[RoutingPolicyRule]
From=185.66.193.40/29
Table=42

Now we can see the ping packets from VM-157 going to the destination and a response coming back to 185.66.193.46 - however, we do not have an ip route to 185.66.193.46 on bb-a directly (without table 42, which is only used for routing with BGP).

Adding this is interesting now:

As we already have a kernel protocol set up for kernel table 42; - we do need another kernel protocol without that directive. Plain adding a new protocol complains at birdc configure check as /etc/bird/bird.conf:1:0 Kernel syncer already attached to table master

So we can not attach the same protocol twice to the same bird table - so we first create a new table in bird on bb-a and add a protocol pipe which syncs the bird state between the master table and the added auxtable:

table auxtable;

protocol pipe {
        table master;
        peer table auxtable;
        export all;
        import all;
}

finally we can add a kernel protocol from auxtable to export our route to 185.66.193.46/32 which was announced from OSPF also on the host main ip routing table itself on bb-a

protocol kernel{
        # export hostroute to default kernel table - needed for communication
        export filter hostroute; # export routes of FFRL ips from bird into kernel routing table
        # Kernel table to synchronize with (default: main)
        table auxtable;
};

A final entering of birdc configure check as well as birdc configure if the former succeeded should then get everything working.

Our ping from VM-157 should reach a destination using the address 185.66.193.46 and return properly.

Troubleshooting

It helps to go through the setup, look into the routes using ip route show as well as ip r s t 42. Check packages using tcpdump, run a traceroute, look into birdc show ospf topology and birdc show route.

I hope that this post makes a lot of the concepts clearer and helps with the understanding of how dynamic routing can be used.