Set up VRRP on a branch HA gateway

Introduction

VRRP is a form of ingress traffic protection. See Introduction to Edge High Availability for more information about ingress and egress traffic protection options.

NetFoundry vCPE gateways can use the VRRP protocol to provide high-availability by way of a virtual IP address that "floats" between multiple instances. The primary gateway instance owns the VIP during normal operations. Should the primary fail, the backup instance will take over the VIP in order to minimize or prevent service down time. The linux package "keepalived" implements the VRRP protocol.

This document covers the following topics:

Failover Framework using VRRP

Keepalived implements the VRRP protocol for failover. The main functionalities provided by this framework are:

  • Failover: The native VRRP protocol purpose, based on a roaming set of VRRP VIPs;
  • VRRP Instance synchronization: We can specify a state monitoring between 2 VRRP Instances, also known as a VRRP sync group. It guarantees that 2 VRRP Instances remain in the same state. The synchronized instances monitor each other;
  • Nice Fallback;
  • Advert Packet integrity: Using IPSEC-AH ICV;
  • System call: During a VRRP state transition, an external script/program may be called.

pastedGraphic_6.png

How VRRP is used on a NetFoundry Gateway

In the picture below a workstation in the left-side private network wants to reach a service provided by the right-side private network. The gateways "NFN endpoint A" and "NFN endpoint B" can reach the desired service in the right-side private network via the NetFoundry Network (NFN). For full redundancy, the two gateways would use separate power circuits, networking hardware, and ISPs.

The two gateways share the VIP 10.0.0.3 via VRRP. Nodes in the left-side private network will use 10.0.0.3 as the next hop to the desired service, via L3 routing or by specifying 10.0.0.3 as the default route via DHCP.

As the designated primary, gateway A will respond to ARP requests for 10.0.0.3. Gateway B will only respond if it determines that the primary gateway is no longer alive. Keepalived establishes a heart beat mechanism between gateway A and gateway B. Should gateway A not be heard from within a specified time period, then gateway B will take over the VIP. In the event that gateway A begins to send heartbeats again, ownership of the VIP is returned to gateway A.

The mechanism for determining if gateway A should stop sending a heart beat extends to a monitor of its internal connection to the NFN. In the event the monitor discovers it is no longer connected to the NFN, it will stop its heartbeat until the connection is restored. The effect is similar to a loss of power on gateway A, which would also stop the heartbeat.

pastedGraphic_1.png

How is it configured?

Keepalived is actively maintained in EPEL repositories of the CentOS7 distribution.  It can be installed by YUM on to each gateway. Keepalived stores its configuration in /etc/keepalived/keepalived.conf, and is unique to the PRIMARY and SECONDARY gateway.  

For example: The following configuration can be applied to the respective gateways.

PRIMARY SECONDARY
vrrp_script chk_nfnalive {
script "/usr/bin/netstat -an |
/usr/bin/egrep 49001.*ESTABLISHED"
interval 2
fall 3
rise 1
}

vrrp_instance NFN {
 state MASTER
 interface [interface_name]
 virtual_router_id 41
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass passw0rd
}

 unicast_src_ip [IP_of_PRIMARY]
 unicast_peer {
  [IP_of_SECONDARY]
 }

 virtual_ipaddress {
[IP_of_VIP] dev ens32
 }

 track_script {
  chk_nfnalive
}

 track_interface {
  [interface_name]
 }
}
vrrp_instance NFN {
 state BACKUP
 interface [interface_name]
 virtual_router_id 41
 priority 149
 advert_int 1
 authentication {
  auth_type PASS
  auth_pass passw0rd
 }

 unicast_src_ip [IP_of_SECONDARY]
 unicast_peer {
  [IP_of_PRIMARY]
 }

 virtual_ipaddress {
  [IP_of_VIP]
 }

 track_interface {
  [interface_name]
 }
}

Installing Keepalived

Install keepalived from the distribution’s repositories or, alternatively, compile from source. Although installing from the repositories is generally the fastest way to get keepalived running on a system, the version of keepalived available in the repositories are typically a few releases behind the latest available stable version.

Installing on Red Hat Enterprise Linux

As of Red Hat 6.4, Red Hat and the clones have included the keepalived package in the base repository. Therefore, run the following to install the keepalived package and all the required dependencies using YUM:

> yum install keepalived

Installing on Debian

Run the following to install the keepalived package and all the required dependencies using Debian’s APT package handling utility:

> apt-get install keepalived

Configuring VRRP

As an example, we can introduce the following LVS topology:

Active/Passive Configuration

pastedGraphic_7.png

Configuring VRRP on Linux

1. Configure static IP-addresses on the primary and backup router (A GW S1 & S2). Do not configure the virtual IP on any of the interfaces. Because the virtual IP-address is not configured on any of the interfaces, Linux will not reply to any packets destined for this IP. This behaviour needs to be changed or VRRP will not work. Edit /etc/sysctl.conf and add this line:

net.ipv4.ip_nonlocal_bind=1

2. Run this command to active this setting:

# sysctl -p

3. Sample configuration of /etc/keepalived/keepalived.conf for MASTER (A GW S1)

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
10.0.0.3
}
}

4. Sample configuration of /etc/keepalived/keepalived.conf for SLAVE (A GW S2)

vrrp_instance VI_1 {
interface eth0
state BACKUP
virtual_router_id 51
priority 100
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
10.0.0.3
}
}

5. Start keepalived:

> sudo service keepalived start

The only configuration difference regarding keepalived between the master and the standby router is the 'priority' setting. The master server should have a higher priority than the backup router (101 vs. 100).

As there can be multiple VRRP configurations active within the same subnet, it is important that you make sure that you set a unique virtual_router_id.

Please do not forget to set your own password in case you enable authentication.

6. Test your VRRP configuration

This is what should happen if the master (A GW S1) is shutdown:

64 bytes from 10.0.0.3: icmp_seq=148 ttl=64 time=0.583 ms
64 bytes from 10.0.0.3: icmp_seq=149 ttl=64 time=0.469 ms
64 bytes from 10.0.0.3: icmp_seq=150 ttl=64 time=0.267 ms
Request timeout for icmp_seq 151
Request timeout for icmp_seq 152
Request timeout for icmp_seq 153
Request timeout for icmp_seq 154
64 bytes from 10.0.0.3: icmp_seq=155 ttl=64 time=0.668 ms
64 bytes from 10.0.0.3: icmp_seq=156 ttl=64 time=0.444 ms
64 bytes from 10.0.0.3: icmp_seq=157 ttl=64 time=0.510 ms

After about five seconds (default) the standby router takes over and starts responding to the virtual IP.

Active/Active Configuration (Simple design)

pastedGraphic_8.png

Configuring VRRP on Linux

1. Configure static IP-addresses on the primary and backup router (A GW S1 & S2). Do not configure the virtual IP on any of the interfaces. In my test environment, I used 10.0.0.1 for the master (A GW S1) and 10.0.0.2 for the backup router (A GW S2).

2. Configure static IP-addresses on the primary and backup router (B GW S1 & S2). Do not configure the virtual IP on any of the interfaces. In my test environment, I used 10.1.0.1 for the master (B GW S1) and 10.1.0.2 for the backup router (B GW S1).

3. Because the virtual IP-address is not configured on any of the interfaces, Linux will not reply to any packets destined for this IP. This behaviour needs to be changed or VRRP will not work. Edit /etc/sysctl.conf and add this line:

net.ipv4.ip_nonlocal_bind=1

4. Run this command to active this setting:

# sysctl -p

5. Sample configuration of /etc/keepalived/keepalived.conf for MASTER (A GW S1) and backup (B GW S1)

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
10.0.0.3
}
}

vrrp_instance VI_2 {
interface eth0
state BACKUP
virtual_router_id 51
priority 100
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
10.1.0.3
}
}

6. Sample configuration of /etc/keepalived/keepalived.conf for SLAVE (A GW S1) and MASTER (B GW S1)

vrrp_instance VI_1 {
interface eth0
state BACKUP
virtual_router_id 51
priority 100
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
10.0.0.3
}
}

vrrp_instance VI_2 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
10.1.0.3
}
}

7. Start keepalived:

> sudo service keepalived start

The only configuration difference regarding keepalived between the master and the standby router is the 'priority' setting. The master server should have a higher priority than the backup router (101 vs. 100).

As there can be multiple VRRP configurations active within the same subnet, it is important that you make sure that you set a unique virtual_router_id.

Please do not forget to set your own password in case you enable authentication.

Active/Active Configuration (Complex design)

To create a virtual LVS director using the VRRPv2 protocol, we define the following architecture:

  • 2 LVS directors in active-active configuration.
  • 4 VRRP Instances per LVS director: 2 VRRP Instance in the MASTER state and 2 in BACKUP state. We use a symmetric state on each LVS directors.
  • 2 VRRP Instances in the same state are to be synchronized to define a persistent virtual routing path.
  • Strong authentication: IPSEC-AH is used to protect our VRRP advertisements from spoofed and reply attacks.

The VRRP Instances are compounded with the following IP addresses:

  • VRRP Instance VI_1: owning VRRIP VIPs VIP1 & VIP2. This instance defaults to the MASTER state on LVS director 1. It stays synchronized with VI_2.
  • VRRP Instance VI_2: owning DIP1. This instance is by default in MASTER state on LVS director 1. It stays synchronized with VI_1.
  • VRRP Instance VI_3: owning VRRIP VIPs VIP3 & VIP4. This instance is in default MASTER state on LVS director 2. It stays synchronized with VI_4.
  • VRRP Instance VI_4: owning DIP2. This instance is in default MASTER state on LVS director 2. It stays synchronized with VI_3.

VRRP Configuration

The whole configuration is done in the /etc/keepalived/keepalived.conf file. In our case study this file on LVS director 1 looks like:

vrrp_sync_group VG1 {
VI_1
VI_2
}

vrrp_sync_group VG2 {
VI_3
VI_4
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
192.168.200.10
192.168.200.11
}
}

vrrp_instance VI_2 {
state MASTER
interface eth1
virtual_router_id 52
priority 150
advert_int 1
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
192.168.100.10
}
}

vrrp_instance VI_3 {
state BACKUP
interface eth0
virtual_router_id 53
priority 100
advert_int 1
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
192.168.200.12
192.168.200.13
}
}

vrrp_instance VI_4 {
state BACKUP
interface eth1
virtual_router_id 54
priority 100
advert_int 1
authentication {
auth_type AH
auth_pass passw0rd
}
virtual_ipaddress {
192.168.100.11
}
}

Then we define the symmetric configuration file on LVS director 2. This means that VI_3 & VI_4 on LVS director 2 are in MASTER state with a higher priority 150 to start with a stable state. Symmetrically VI_1 & VI_2 on LVS director 2 are in default BACKUP state with lower priority of 100. | This configuration file specifies 2 VRRP Instances per physical NIC. When you run Keepalived on LVS director 1 without running it on LVS director 2, LVS director 1 will own all the VRRP VIP. So if you use the ip utility you may see something like: (On Debian the ip utility is part of iproute):

> sudo ip address list
1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:5e:00:01:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.5/24 brd 192.168.200.255 scope global eth0
    inet 192.168.200.10/32 scope global eth0
    inet 192.168.200.11/32 scope global eth0
    inet 192.168.200.12/32 scope global eth0
    inet 192.168.200.13/32 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:5e:00:01:32 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.5/24 brd 192.168.201.255 scope global eth1
    inet 192.168.100.10/32 scope global eth1
    inet 192.168.100.11/32 scope global eth1

Then simply start Keepalived on the LVS director 2 and you will see:

> sudo ip address list
1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:5e:00:01:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.5/24 brd 192.168.200.255 scope global eth0
    inet 192.168.200.10/32 scope global eth0
    inet 192.168.200.11/32 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:5e:00:01:32 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.5/24 brd 192.168.201.255 scope global eth1
    inet 192.168.100.10/32 scope global eth1

Symmetrically on LVS director 2 you will see:

> sudo ip address list
1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:5e:00:01:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.5/24 brd 192.168.200.255 scope global eth0
    inet 192.168.200.12/32 scope global eth0
    inet 192.168.200.13/32 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:5e:00:01:32 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.5/24 brd 192.168.201.255 scope global eth1
    inet 192.168.100.11/32 scope global eth1

The VRRP VIPs are:

  • VIP1 = 192.168.200.10
  • VIP2 = 192.168.200.11
  • VIP3 = 192.168.200.12
  • VIP4 = 192.168.200.13
  • DIP1 = 192.168.100.10
  • DIP2 = 192.168.100.11

The use of VRRP keyword “sync_instance” imply that we have defined a pair of MASTER VRRP Instance per LVS directors ó (VI_1,VI_2) & (VI_3,VI_4). This means that if eth0 on LVS director 1 fails then VI_1 enters the MASTER state on LVS director 2 so the MASTER Instance distribution on both directors will be: (VI_2) on director 1 & (VI_1,VI_3,VI_4) on director 2. We use “sync_instance” so VI_2 is forced to BACKUP the state on LVS director 1. The final VRRP MASTER instance distribution will be: (none) on LVS director 1 & (VI_1,VI_2,VI_3,VI_4) on LVS director 2. If eth0 on LVS director 1 became available, the distribution will transition back to the initial state.

Using VRRP with Virtual MAC Address

To reduce takeover impact, some networking environment would require using VRRP with VMAC address. To reach that goal Keepalived VRRP framework implements VMAC support by the invocation of ‘use_vmac’ keyword in configuration file.

Internally, Keepalived code will bring up virtual interfaces, each interface dedicated to a specific virtual_router. Keepalived uses Linux kernel macvlan driver to defines thoses interfaces. It is then mandatory to use kernel compiled with macvlan support.

In addition, we can mention that VRRP VMAC will work only with kernel including the following patch:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git;a=commitdiff;h=729e72a10930ef765c11a5a35031ba47f18221c4

By default, MACVLAN interface are in VEPA mode which filters out received packets whose MAC source address matches that of the MACVLAN interface. Setting MACVLAN interface in private mode will not filter based on source MAC address.

Alternatively, you can specify ‘vmac_xmit_base’ which will cause the VRRP messages to be transmitted and received on the underlying interface whilst ARP will happen from the the VMAC interface.

You may also need to tweak your physical interfaces to play around with well known ARP issues. If you have issues, try the following configurations:

Global configuration:

net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 1
net.ipv4.conf.all.arp_filter = 0

Physical interface configuration

For the physical ethernet interface running VRRP instance use:

net.ipv4.conf.eth0.arp_filter = 1

VMAC interface

Consider the following VRRP configuration:

vrrp_instance instance1 {
state BACKUP
interface eth0
virtual_router_id 250
use_vmac
vmac_xmit_base
priority 150
advert_int 1
virtual_ipaddress {
10.0.0.254
}
}

The use_vmac keyword will drive keepalived code to create a macvlan interface named vrrp.250(default internal paradigm is vrrp.{virtual_router_id}, you can override this naming by giving an argument to ‘use_vmac’ keyword, eg: use_vmac vrrp250).

You then need to configure interface with:

net.ipv4.conf.vrrp.250.arp_filter = 0
net.ipv4.conf.vrrp.250.accept_local = 1
net.ipv4.conf.vrrp.250.rp_filter = 0

You can create notify_master script to automate this configuration step for you:

vrrp_instance instance1 {
state BACKUP
interface eth0
virtual_router_id 250
use_vmac
priority 150
advert_int 1
virtual_ipaddress {
10.0.0.254
}
notify_master "/usr/local/bin/vmac_tweak.sh vrrp.250"
}

 

 

Was this article helpful?
0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.