Forwarding in vPC through FHRP backup

In this post, we will cover the benefits introduced by Virtual Port-Channel (vPC) for First Hop Redundancy Protocols (FHRP).

Hot Standby Router Protocol (HSRP) and Virtual Router Redundancy Protocol (VRRP) are two of these FHRPs.

HSRP is Cisco proprietary whereas VRRP is a standard-based protocol and allows interoperability between different vendors.

vPC enhances FHRPs by allowing them to operate in active-active mode at data plane while at control plane the FHRP still works in active-backup mode.

The above statement considers active as the ability to forward traffic and it is just a coincidence with the HSRP state. In VRRP, the master forwards the traffic, whereas the backup does not.

Coming back to the vPC use case, at the control plane, the active router is still responsible to reply to ARP Requests with the virtual MAC address. In case the ARP request is received by the backup router, the ARP request is forwarded to the active router through the VPC peer-link which then replies with the virtual MAC address.

The feature that allows FHRPs to work in active-active mode at data plane is vPC built-in and there is no additional configuration required to enable it.

The active-active operation mode is facilitated by imposing G bit (Gateway bit) for the FHRP virtual MAC in the MAC address table on both vPC peer devices. We are going to see what it means later.

Let’s see an example using VRRP as FHRP on two Nexus devices.

This is the network diagram:

R_ACCESS has a default gateway 172.16.123.254 which is the virtual IP address configured in VRRP.

R_ACCESS#show ip route

Gateway of last resort is 172.16.123.254 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 172.16.123.254
 172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
 C 172.16.123.0/24 is directly connected, GigabitEthernet0/1
 L 172.16.123.100/32 is directly connected, GigabitEthernet0/1
 R_ACCESS#

The ARP table on R_ACCESS shows the virtual MAC address for the default gateway IP:

R_ACCESS#show ip arp
 Protocol Address Age (min) Hardware Addr Type Interface
 Internet 172.16.123.254 0 0000.5e00.0101 ARPA GigabitEthernet0/1
 Internet 172.16.123.100 - fa16.3e8b.ce06 ARPA GigabitEthernet0/1
 R_ACCESS#

A best practice recommendation is to have at control-plane the active router the same device that is the vPC primary peer device.

In this case, NX_OS-1 is the master router and in the same time it is the primary peer.

NX_OS-1# show vrrp detail

Vlan1000 - Group 1 (IPV4)
 State is Master
 Virtual IP address is 172.16.123.254
 Priority 110, Configured 110
 Forwarding threshold(for VPC), lower: 1 upper: 110
 Advertisement interval 1
 Preemption enabled
 Authentication text "VRRP_AU"
 Virtual MAC address is 0000.5e00.0101
 Master router is Local
NX_OS-1# show vpc
 Legend:
 (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 21
 Peer status : peer adjacency formed ok
 vPC keep-alive status : peer is alive
 Configuration consistency status : success
 Per-vlan consistency status : success
 Type-2 consistency status : success
 vPC role : primary
 Number of vPCs configured : 1
 Peer Gateway : Disabled
 Dual-active excluded VLANs : -
 Graceful Consistency Check : Enabled
 Auto-recovery status : Disabled
 Delay-restore status : Timer is off.(timeout = 30s)
 Delay-restore SVI status : Timer is off.(timeout = 10s)
 Operational Layer3 Peer-router : Disabled

vPC Peer-link status
 ---------------------------------------------------------------------
 id Port Status Active vlans
 -- ---- ------ -------------------------------------------------
 1 Po21 up 1000

vPC status
 ----------------------------------------------------------------------------
 Id Port Status Consistency Reason Active vlans
 -- ------------ ------ ----------- ------ ---------------
 1 Po1 up success success 1000

Please check “show vpc consistency-parameters vpc” for the consistency reason of down vpc and for type-2 consistency reasons for any vpc.

NX_OS-1#

The Port-Channel between the access switch and the two Nexus devices is up:

SW_ACCESS#show etherchannel summary
 Flags: D - down P - bundled in port-channel
 I - stand-alone s - suspended
 H - Hot-standby (LACP only)
 R - Layer3 S - Layer2
 U - in use N - not in use, no aggregation
 f - failed to allocate aggregator

M - not in use, minimum links not met
 m - not in use, port not aggregated due to minimum links not met
 u - unsuitable for bundling
 w - waiting to be aggregated
 d - default port

A - formed by Auto LAG

Number of channel-groups in use: 1
 Number of aggregators: 1

Group Port-channel Protocol Ports
 ------+-------------+-----------+-----------------------------------------------
 1 Po1(SU) LACP Gi0/1(P) Gi0/2(P)

SW_ACCESS#

R_CORE is running OSPF with the two Nexus devices for end-to-end reachability:

R_CORE#show ip route
 172.16.0.0/24 is subnetted, 1 subnets
 O 172.16.123.0 [110/41] via 10.10.13.2, 00:01:57, GigabitEthernet0/2
 [110/41] via 10.10.12.2, 00:03:30, GigabitEthernet0/1
 R_CORE#

This being said, when the traffic from R_ACCESS needs to go R_CORE (that is, pass through one of the two Nexus devices), it will reach the access switch. At this point, the switch will decide based on the hashing algorithm which of the two links from the Port-Channel will use. It can go directly to the master VRRP router or it can go to the backup VRRP router.

The below diagram shows the situation when the traffic hits NX_OS-1 which is the master VRRP:

On R_CORE device, there is an access list applied on each the two interfaces matching and logging the traffic from R_ACCESS to R_CORE’s loopback IP address to confirm that traffic is received for this pair (172.16.123.100,1.1.1.1) on both interfaces:

interface GigabitEthernet0/1
 ip address 10.10.12.1 255.255.255.0
 ip access-group VIA_NX_OS-1 in
 duplex auto
 speed auto
 media-type rj45
 !
 interface GigabitEthernet0/2
 ip address 10.10.13.1 255.255.255.0
 ip access-group VIA_NX_OS-2 in
 duplex auto
 speed auto
 media-type rj45
 !
 ip access-list extended VIA_NX_OS-1
 permit icmp host 172.16.123.100 host 1.1.1.1 log-input
 permit ip any any
 ip access-list extended VIA_NX_OS-2
 permit icmp host 172.16.123.100 host 1.1.1.1 log-input
 permit ip any any

When traffic takes the path through NX_OS-1, the packet matches the ACL from GigabitEthernet0/1:

%SEC-6-IPACCESSLOGDP: list VIA_NX_OS-1 permitted icmp 172.16.123.100
(GigabitEthernet0/1 5e00.0000.0007) -> 1.1.1.1 (0/0), 5 packets

And the ACL counter is increasing:

R_CORE#show ip access-lists
 Extended IP access list VIA_NX_OS-1
 10 permit icmp host 172.16.123.100 host 1.1.1.1 log-input (5 matches)
 20 permit ip any any (16 matches)
 Extended IP access list VIA_NX_OS-2
 10 permit icmp host 172.16.123.100 host 1.1.1.1 log-input
 20 permit ip any any (8 matches)
 R_CORE#

The diagram below shows when the traffic hits NX_OS-2:

When the traffic takes this path, the ACL matches the packet and logs it:

%SEC-6-IPACCESSLOGDP: list VIA_NX_OS-2 permitted icmp 172.16.123.100
(GigabitEthernet0/2 5e00.0001.0007) -> 1.1.1.1 (0/0), 5 packets

Again, the ACL counters are increasing:

R_CORE#show ip access-lists
 Extended IP access list VIA_NX_OS-1
 10 permit icmp host 172.16.123.100 host 1.1.1.1 log-input (5 matches)
 20 permit ip any any (25 matches)
 Extended IP access list VIA_NX_OS-2
 10 permit icmp host 172.16.123.100 host 1.1.1.1 log-input (5 matches)
 20 permit ip any any (8 matches)
 R_CORE#

As you can, see whenever the traffic goes to NX_OS-2, the traffic is routed directly to the destination according to the routing table.

This is how the ARP/MAC table looks on NX_OS-1, the master VRRP:

NX_OS-1# show ip arp

Flags: * - Adjacencies learnt on non-active FHRP router
 + - Adjacencies synced via CFSoE
 # - Adjacencies Throttled for Glean
 CP - Added via L2RIB, Control plane Adjacencies
 PS - Added via L2RIB, Peer Sync
 RO - Dervied from L2RIB Peer Sync Entry
 D - Static Adjacencies attached to down interface

IP ARP Table for context default
 Total number of entries: 3
 Address Age MAC Address Interface Flags
 10.10.12.1 00:01:22 fa16.3ecd.b31a Ethernet1/4
 172.16.123.2 00:00:09 5e00.0001.0007 Vlan1000
 172.16.123.254 - 0000.5e00.0101 Vlan1000
 NX_OS-1# show mac address-table
 Legend:
 * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
 age - seconds since last seen,+ - primary entry using vPC Peer-Link,
 (T) - True, (F) - False, C - ControlPlane MAC
 VLAN MAC Address Type age Secure NTFY Ports
 ---------+-----------------+--------+---------+------+----+------------------
 G 1000 0000.5e00.0101 static - F F sup-eth1(R)
 G - 5e00.0000.0007 static - F F sup-eth1(R)
 G 1000 5e00.0000.0007 static - F F sup-eth1(R)
 * 1000 5e00.0001.0007 static - F F vPC Peer-Link(R)
 NX_OS-1#

And on NX_OS-2:

NX_OS-2# show ip arp

Flags: * - Adjacencies learnt on non-active FHRP router
 + - Adjacencies synced via CFSoE
 # - Adjacencies Throttled for Glean
 CP - Added via L2RIB, Control plane Adjacencies
 PS - Added via L2RIB, Peer Sync
 RO - Dervied from L2RIB Peer Sync Entry
 D - Static Adjacencies attached to down interface

IP ARP Table for context default
 Total number of entries: 3
 Address Age MAC Address Interface Flags
 10.10.13.1 00:15:50 fa16.3ee0.57bf Ethernet1/4
 172.16.123.1 00:01:18 5e00.0000.0007 Vlan1000
 172.16.123.254 00:00:05 0000.5e00.0101 Vlan1000
 NX_OS-2# show mac address-table
 Legend:
 * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
 age - seconds since last seen,+ - primary entry using vPC Peer-Link,
 (T) - True, (F) - False, C - ControlPlane MAC
 VLAN MAC Address Type age Secure NTFY Ports
 ---------+-----------------+--------+---------+------+----+------------------
 G 1000 0000.5e00.0101 static - F F vPC Peer-Link(R)
 * 1000 5e00.0000.0007 static - F F vPC Peer-Link(R)
 G - 5e00.0001.0007 static - F F sup-eth1(R)
 G 1000 5e00.0001.0007 static - F F sup-eth1(R)
 NX_OS-2#

On the active router at control plane, the virtual MAC points to sup-eth1(R), while on the backup router at control plane, the virtual MAC points to the vPC peer-link and this is a good way to determine which router is the active and which is the backup on control plane level.

As mentioned at the beginning of the article, you do not need to do anything to benefit from this mode of operation of FHRP in combination with vPCs.

Reference:
1. Design and Configuration Guide: Best Practices for Virtual Port Channels (vPC) on Cisco Nexus 7000 Series Switches

Thank you to Paris Arau for his contributions to this article.