VXLAN with MP-BGP EVPN control plane
Notice: This blog post was originally published on Indeni before its acquisition by BlueCat.
The content reflects the expertise and perspectives of the Indeni team at the time of writing. While some references may be outdated, the insights remain valuable. For the latest updates and solutions, explore the rest of our blog
This is the second part of a series covering VXLAN on NEXUS devices using Multi-Protocol BGP(MP-BGP) as control plane. The first part looked at unicast and multicast control-plane.
Some of the drawbacks of these options were mentioned in the first part.
The advantage of Ethernet VPN(EVPN) is that the MAC learning happens in control-plane rather than in data plane. Data plane learning means that the MAC address is learned from an Ethernet frame that reach the device (this is no different than the traditional MAC learning).
The advantage of control-plane MAC learning is that it offers greater control over the MAC learning providing granularity for who can learn what by applying policies
The EVPN NLRI is carried in BGP using Address Family Identifier (AFI) of 25 (L2VPN) and a Subsequent Address Family Identifier (SAFI) of 70 (EVPN).
There is quite complexity involved in EVPN which is outside the scope of this series. The purpose of this series is to show how you can configure a specific features and how to confirm that is working properly.
The following is the network topology, very similar to where we left part 1:
The VXLAN/EVPN configuration is built on top of VXLAN with multicast control-plane, that is, the following configuration is applied on the setup that we left at the end of the first part.
The next step is to enable EVPN and BGP on all devices:
NX_OS_4# show running-config | include "bgp|evpn" nv overlay evpn feature bgp NX_OS_4# BGP configuration on leafs is this: NX_OS_1# show running-config bgp !Command: show running-config bgp !Time: Thu Dec 14 17:58:54 2017 version 7.0(3)I6(1) feature bgp router bgp 65100 neighbor 1.1.1.4 remote-as 65100 update-source loopback0 address-family l2vpn evpn send-community send-community extended evpn vni 10100 l2 rd auto route-target import auto route-target export auto NX_OS_1#
And on the spine. NX_OS_4 will be the route reflector so that BGP updates will be reflected to all other BGP speakers:
NX_OS_4# show running-config bgp !Command: show running-config bgp !Time: Thu Dec 14 17:59:22 2017 version 7.0(3)I6(1) feature bgp router bgp 65100 neighbor 1.1.1.1 remote-as 65100 update-source loopback0 address-family l2vpn evpn send-community send-community extended route-reflector-client neighbor 1.1.1.2 remote-as 65100 update-source loopback0 address-family l2vpn evpn send-community send-community extended route-reflector-client neighbor 1.1.1.3 remote-as 65100 update-source loopback0 address-family l2vpn evpn send-community send-community extended route-reflector-client NX_OS_4#
In addition to this, “host-reachability protocol bgp” will configure BGP as the method for host reachability advertisement.
interface nve1 no shutdown source-interface loopback0 host-reachability protocol bgp member vni 10100 mcast-group 226.0.0.100
Once this is in place, BGP sessions should be up:
NX_OS_4(config-router)# show bgp all summary BGP summary information for VRF default, address family IPv4 Unicast BGP summary information for VRF default, address family IPv6 Unicast BGP summary information for VRF default, address family L2VPN EVPN BGP router identifier 1.1.1.4, local AS number 65100 BGP table version is 23, L2VPN EVPN config peers 3, capable peers 3 0 network entries and 0 paths using 0 bytes of memory BGP attribute entries [0/0], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 1.1.1.1 4 65100 47 50 0 0 0 00:00:02 0 1.1.1.2 4 65100 48 49 0 0 0 00:00:02 0 1.1.1.3 4 65100 46 50 0 0 0 00:00:07 0 NX_OS_4(config-router)#
Now the learning for this VXLAN segment is control-plane and not data-plane as in the unicast/multicast control-plane:
NX_OS_1# show nve vni Codes: CP - Control Plane DP - Data Plane UC - Unconfigured SA - Suppress ARP Interface VNI Multicast-group State Mode Type [BD/VRF] Flags --------- -------- ----------------- ----- ---- ------------------ ----- nve1 10100 226.0.0.100 Up CP L2 [100] NX_OS_1#
Now, the MAC address table is empty on all devices:
NX_OS_1# show system internal l2fwder mac Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ NX_OS_1#
Packet captures where started on various points in the network: on e1/2 and e1/1 on NX_OS_1, on e1/2 and e1/3 on NX_OS_4. Basically, the traffic between leafs and spine is captured like this along with the traffic to/from R1.
Just before any ping test, R1 sent a Gratuitous ARP(independently from our testing):
This was relayed further to the network using multicast IP address as destination IP by NX_OS_1:
This GARP reached NX_OS_2 and NX_OS_3. This is from NX_OS_2:
But what happens after the GARP was received by NX_OS_1 is where EVPN comes into play. NX_OS_1 sends a BGP update to NX_OS_4(the route reflector) announcing the MAC address received from R1:
Next, NX_OS_4 sends a BGP update to its route reflector clients. This is BGP update received by NX_OS_2:
And this is the BGP update received by NX_OS_3:
In this moment, NX_OS_2 and NX_OS_3 know about R1’s MAC address and it was learned via control-plane.
Moments later, R2 sends a similar GARP triggering BGP Updates that reach NX_OS_1 and NX_OS_3(the same process described above repeats).
A ping started from R1 to R2 should be successful:
R1(config-if)#do ping 100.100.100.2 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 100.100.100.2, timeout is 2 seconds: .!!!! Success rate is 80 percent (4/5), round-trip min/avg/max = 19/19/21 ms R1(config-if)# do sh arp Protocol Address Age (min) Hardware Addr Type Interface Internet 100.100.100.1 - fa16.3e6e.a47b ARPA GigabitEthernet0/1 Internet 100.100.100.2 6 fa16.3ebb.35e8 ARPA GigabitEthernet0/1 R1(config-if)#
So, let us see what is happening.
R1 will send an ARP Request to resolve R2 IP address which in turn will be encapsulated by NX_OS_1 and sent to the multicast group:
This ARP Request reach NX_OS_2 and NX_OS_3, but only NX_OS_2 replies because R2 is behind NX_OS_2.
After the ARP resolution, ping starts to work:
Let us see the content of the MAC address table on NX_OS_1. It should show the MAC address of R1 and R2:
NX_OS_1# show system internal l2fwder mac Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ * 100 fa16.3e6e.a47b dynamic 00:00:19 F F Eth1/2 * 100 fa16.3ebb.35e8 static - F F (0x47000001) nve-peer1 1.1.1.2 NX_OS_1#
Observe that the MAC address of R2 is shown as static and this is how it is when the MAC address is learned via EVPN.
It is similar on NX_OS_2:
NX_OS_2# show system internal l2fwder mac Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ * 100 fa16.3e6e.a47b static - F F (0x47000001) nve-peer1 1.1.1.1 * 100 fa16.3ebb.35e8 dynamic 00:00:28 F F Eth1/2 NX_OS_2#
But on NX_OS_3, looks like. NX_OS_3 populates it MAC address table from BGP updates:
NX_OS_3(config-if)# show system internal l2fwder mac Legend: * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC age - seconds since last seen,+ - primary entry using vPC Peer-Link, (T) - True, (F) - False, C - ControlPlane MAC VLAN MAC Address Type age Secure NTFY Ports ---------+-----------------+--------+---------+------+----+------------------ * 100 fa16.3e6e.a47b static - F F (0x47000001) nve-peer1 1.1.1.1 * 100 fa16.3ebb.35e8 static - F F (0x47000002) nve-peer2 1.1.1.2 NX_OS_3(config-if)#
NX_OS_3 has two peers learned via EVPN:
NX_OS_3(config-if)# show nve peers Interface Peer-IP State LearnType Uptime Router-Mac --------- --------------- ----- --------- -------- ----------------- nve1 1.1.1.1 Up CP 00:03:34 n/a nve1 1.1.1.2 Up CP 00:03:32 n/a NX_OS_3(config-if)#
Whereas NX_OS_1 and NX_OS_2 have only one, each other. This is because NX_OS_3 did not send any BGP update. This is from NX_OS_1 and it is similar on NX_OS_2:
NX_OS_1# show nve peers detail Details of nve Peers: ---------------------------------------- Peer-Ip: 1.1.1.2 NVE Interface : nve1 Peer State : Up Peer Uptime : 00:03:08 Router-Mac : n/a Peer First VNI : 10100 Time since Create : 00:03:08 Configured VNIs : 10100 Provision State : add-complete Route-Update : Yes Peer Flags : DisableLearn Learnt CP VNIs : 10100 Peer-ifindex-resp : Yes ---------------------------------------- NX_OS_1#
You can see the content of the EVPN database(looks similar on NX_OS_2:
NX_OS_1# show bgp l2vpn evpn BGP routing table information for VRF default, address family L2VPN EVPN BGP table version is 19, local router ID is 1.1.1.1 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 1.1.1.1:32867 (L2VNI 10100) *>l[2]:[0]:[0]:[48]:[fa16.3e6e.a47b]:[0]:[0.0.0.0]/216 1.1.1.1 100 32768 i *>i[2]:[0]:[0]:[48]:[fa16.3ebb.35e8]:[0]:[0.0.0.0]/216 1.1.1.2 100 0 i Route Distinguisher: 1.1.1.2:32867 *>i[2]:[0]:[0]:[48]:[fa16.3ebb.35e8]:[0]:[0.0.0.0]/216 1.1.1.2 100 0 i NX_OS_1#
And on NX_OS_3:
NX_OS_3(config-if)# show bgp l2vpn evpn BGP routing table information for VRF default, address family L2VPN EVPN BGP table version is 20, local router ID is 1.1.1.3 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i njected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 1.1.1.1:32867 *>i[2]:[0]:[0]:[48]:[fa16.3e6e.a47b]:[0]:[0.0.0.0]/216 1.1.1.1 100 0 i Route Distinguisher: 1.1.1.2:32867 *>i[2]:[0]:[0]:[48]:[fa16.3ebb.35e8]:[0]:[0.0.0.0]/216 1.1.1.2 100 0 i Route Distinguisher: 1.1.1.3:32867 (L2VNI 10100) *>i[2]:[0]:[0]:[48]:[fa16.3e6e.a47b]:[0]:[0.0.0.0]/216 1.1.1.1 100 0 i *>i[2]:[0]:[0]:[48]:[fa16.3ebb.35e8]:[0]:[0.0.0.0]/216 1.1.1.2 100 0 i NX_OS_3(config-if)#
And this would be pretty much about what you need to configure on Nexus devices have VXLAN/EVPN and how to check that is working as expected.
This part also concludes the VXLAN series that covered different control-plane mechanisms, unicast, multicast and EVPN.
Reference:
1. Ethernet VPN – What’s the big deal about it?
2. A Summary of Cisco VXLAN Control Planes: Multicast, Unicast, MP-BGP EVPN
3. BGP MPLS-Based Ethernet VPN
4. Configuring VXLAN BGP EVPN
Thank you to Paris Arau for his contributions to this article.
Key takeawaysThis key takeaway was generated through LLMs crawling the page and coming up with an overview of the content.This article, the second part of a VXLAN on Nexus series, demonstrates using MP-BGP EVPN as the control plane so MAC learning happens in the control plane rather than from data-plane frames. It describes enabling EVPN and BGP with a route reflector, using the BGP host-reachability method, and shows how Gratuitous ARP triggers EVPN BGP updates that populate MAC/EVPN tables across spines and leafs, impacting operational visibility and control. The outcome confirms that EVPN provides centralized MAC advertisement, static-appearing MAC entries learned via BGP, and procedures to verify correct VXLAN/EVPN operation on Nexus devices.
Why use EVPN with MP-BGP instead of unicast or multicast control planes for VXLAN on Nexus devices?
EVPN with MP-BGP moves MAC learning into the control plane, meaning MAC addresses are advertised via BGP (AFI 25 / SAFI 70) rather than learned from traffic in the data plane. This provides greater operational control and policy granularity over who can learn which MACs, avoids reliance on flooding or multicast for discovery, and produces stable, static-appearing MAC table entries on devices. In the described topology EVPN allowed Gratuitous ARP-originated MACs to be distributed via BGP through a route reflector, improving consistency and visibility across leaf and spine devices compared with unicast or multicast-only approaches.
How was host reachability configured and validated in the lab topology?
The lab configured BGP as the host-reachability protocol (host-reachability protocol bgp) and enabled EVPN on all Nexus devices, with NX_OS_4 acting as a route reflector so updates propagate to all BGP speakers. Validation included bringing BGP sessions up, observing empty MAC tables before EVPN advertisement, capturing packets on leaf-to-spine interfaces, and watching Gratuitous ARP (GARP) events trigger BGP EVPN NLRI updates. Those BGP updates were seen on the route reflector and on route-reflector clients (NX_OS_2 and NX_OS_3), after which MAC tables populated and pings between hosts succeeded, confirming reachability via control-plane learning.
What observable differences appear in device MAC tables and EVPN databases after EVPN learning?
After EVPN learning, MAC tables show entries learned via control-plane and often appear static; for example NX_OS_1 and NX_OS_2 displayed R1 and R2 MACs with static-like attributes because they were learned from BGP updates. NX_OS_3 populated its MAC table from received BGP EVPN updates and showed two peers learned via EVPN because it did not originate a BGP update for that MAC, while NX_OS_1 and NX_OS_2 each showed one peer. The EVPN database on each device reflected the EVPN NLRI advertisements, demonstrating that MAC/IP endpoint information is stored and distributed via BGP rather than by traditional data-plane learning.
Related content
How to map your network with user-defined links in Integrity X
Map your network with user-defined links in Integrity X to define and manage custom relationships, such as dual-stack and NAT environments.
Automate your DDI modernization path by migrating with Micetro
Automate cross-platform DNS and DHCP migration with Micetro to reduce risk, eliminate manual effort, and modernize infrastructure faster.
Your journey to intelligent NetOps begins at Cisco Live
Visit BlueCat’s booth or book a meeting now to learn more about how our solutions can help you build a network that supports constant change.
Replace BIND and ISC with Micetro DNS/DHCP Server (MDDS)
Tired of patching and manually configuring BIND DNS and ISC DHCP? Discover how Micetro MDDS appliances can replace them for modern DDI.