IBGP with Linux and pfSense

Posted on Wed 14 June 2017 in Linux

We have two servers with Ubuntu 16.04 that will use bird to provide BGP routing:

  1. lb1 - IP 10.0.1.201
  2. lb2 - IP 10.0.1.202

Our anycast IP will be 10.0.7.1.

The pfSense server will use openbgpd to provide BGP routing and its IP is 10.0.1.1.

Configuring openbgpd on pfSense

First you need to add the openbgpd package, you can do this through System -> Packages -> Available Packages.

When the package is installed, go to Services -> Openbgpd -> Raw config and enter the below:

AS 65001
holdtime 12
network inet connected
router-id 10.0.1.1

group "loadbalancers" {
    remote-as 65001
    neighbor 10.0.1.201 {
        descr "lb1"
    }
    neighbor 10.0.1.202 {
        descr "lb2"
    }
}

allow quick from group loadbalancers
allow quick to group loadbalancers
deny from any
deny to any

What is all this? First of all we set our AS to 65001 which is in the private range (64512-65534), meant to be used in internal networks like ours. Then we set the holdtime to 12 seconds, this is the "timeout" for BGP neighbors which means if we haven't received a keepalive message from one of the neighbors in 12 seconds we drop their session.

The "network inet connected" line means we will announce routes to networks we are directly connected to. "router-id" is the IP address our neighbors can reach us on.

The group is self-explanatory I hope and towards the end of the file we have the ACL section controlling who we accept routes from and who we will pass on routes to.

Note the importance of the "quick" option here, if we drop it our configuration will cease to work unless we reorder the lines to have the "deny" lines first, followed by the "allow" lines.

Go to Status -> Services and start the openbgpd service

Prep the Ubuntu servers

Install the bird package:

sudo apt-get install bird -y

Add the anycast IP address to the loopback interface:

sudo ip addr add 10.0.7.1/32 dev lo

Ensure your configuration for the "lo" interface looks as below in /etc/network/interfaces:

auto lo
iface lo inet loopback

iface lo inet static
    address 10.0.7.1
    netmask 255.255.255.255

Verify that your loopback interface is correctly configured:

ip addr show dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
    inet 10.0.7.1/32 scope global lo
        valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever

Configure bird

On both your Ubuntu servers configure bird in /etc/bird/bird.conf:

listen bgp address SERVERIP;
router id SERVERIP;

filter anycastip {
    if net = 10.0.7.1/32 then accept;
    reject;
}

protocol bgp {
    local as 65001;
    neighbor 10.0.1.1 as 65001;
    export filter anycastip;
    import filter anycastip;
    source address SERVERIP;
}

protocol device {
    scan time 60;
}

protocol direct {
    interface "lo";
}

Replace "SERVERIP" with each Ubuntu server's IP address, in our example 10.0.1.201 for the first server and 10.0.1.202 for the second server.

In short, we listen/identify ourselves with our IP address and we create a filter that will match our anycast IP and ignore/drop all other routes announced over BGP. The only neighbor needed here is our closest FW/router (pfSense in our case) which will ensure traffic from the rest of the network can find the anycast IP.

We need the "protocol device" section to get information from the available network interfaces, where for instance "lo" contains our anycast IP.

The "protocol direct" section is necessary to generate routes for all the directly connected networks, by default it does that for all interfaces and while that is not an issue in our case it is still nice to limit it to the only interface we are interested in, the "lo" interface.

Start the service:

sudo service bird start

Verify that it is working:

sudo birdc

BIRD 1.5.0 ready.
bird> show protocols all bgp1
name     proto    table    state  since       info
bgp1     BGP      master   up     14:30:20    Established
  Preference:     100
  Input filter:   anycastip
  Output filter:  anycastip
  Routes:         0 imported, 1 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:             12          0         12          0          0
    Import withdraws:            6          0        ---         18          0
    Export updates:              1          0          0        ---          1
    Export withdraws:            0        ---        ---        ---          0
  BGP state:          Established
    Neighbor address: 10.0.1.1
    Neighbor AS:      65001
    Neighbor ID:      10.0.1.1
    Neighbor caps:    refresh restart-able AS4
    Session:          internal multihop AS4
    Source address:   10.0.1.220
    Hold timer:       8/12
    Keepalive timer:  2/4

The key pieces here are the "Routes" statement which tells us we have exported 1 route which is the route to 10.0.7.1, our anycast IP - the other important piece is the "BGP state" statement which tells us the connection is "Established". All green here!

Finally

You should now be able to access the two Ubuntu servers through the anycast IP 10.0.7.1 from other computers in your network, easily verified with ping.

Also a good idea to take a look at BGP on the pfSense server:

bgpctl show rib neighbor lb1

flags: * = Valid, > = Selected, I = via IBGP, A = Announced, S = Stale
origin: i = IGP, e = EGP, ? = Incomplete

flags destination          gateway          lpref   med aspath origin
I*>   10.0.7.1/32          10.0.1.220         100     0 i

BGP Linux pfSense Ubuntu