EVPN-VXLAN: The Modern Datacenter Fabric

There’s a particular kind of pain that comes with trying to scale a datacenter that was designed around a three-tier hierarchy and Spanning Tree Protocol. Not the dramatic, pager-at-2am kind of pain, but the slow, grinding kind where every growth project turns into a conversation about VLAN limits, where your spanning tree topology is so convoluted nobody fully trusts it anymore, and where adding redundancy in the wrong place somehow makes the network less stable rather than more. If you’ve managed a legacy datacenter fabric through a period of real growth, you know exactly what that feels like.

The three-tier model, core, aggregation, and access, made a lot of sense for a long time. It was borrowed from campus networking, and for the workloads that dominated datacenters in the early 2000s, it worked well enough. Traffic was mostly north-south: clients outside the datacenter talking to servers inside it. The hierarchy was optimized for exactly that flow, and the aggregation layer was where all the interesting routing happened. Spanning Tree kept the loops out of the Layer 2 domains at the access layer, and everyone got used to the idea that redundant uplinks meant blocked ports and wasted bandwidth.

What killed the three-tier model wasn’t a single design flaw. It was a shift in how applications actually communicate. As virtualization took hold and applications moved toward distributed, multi-tier architectures where web servers talk to app servers talk to database clusters, the traffic pattern flipped. East-west traffic, server-to-server communication inside the datacenter, started to dominate. And a hierarchy built for north-south flows, with Spanning Tree blocking half your bandwidth and Layer 2 domains that couldn’t stretch cleanly across the fabric, was badly suited for it. You could feel it in the workarounds: hacks to keep VLANs consistent across aggregation layers, convoluted STP tuning to keep specific paths active, proprietary extensions like vPC and VSS to make a pair of switches look like one to keep STP from getting in the way entirely.

EVPN-VXLAN didn’t appear as a clean replacement for all of that overnight. It came together from a few separate problems being solved in parallel, and understanding those pieces individually makes the overall architecture a lot easier to reason about.

The physical foundation: why leaf-spine replaced the hierarchy

Before getting into the overlay, it’s worth spending a moment on the underlay, because the shift to leaf-spine Clos topology is what made everything else possible. The design traces back to Charles Clos’s 1953 work on multistage telephone switching networks, and the core insight is simple: if every leaf switch connects to every spine switch, you get a fabric that is non-blocking by design, and adding capacity in either direction is straightforward and predictable.

In a standard two-tier leaf-spine fabric, every server connects to a leaf, and every leaf connects to every spine. There are no connections between leaves, and no connections between spines. Traffic between two servers on different leaves always traverses exactly two hops: leaf to spine to leaf. That consistency matters enormously for latency predictability, and it means the fabric has no concept of a preferred path or a blocked port. Every leaf-to-spine link is active and load-balanced simultaneously via ECMP. Blocking redundant ports to prevent loops, which was Spanning Tree’s entire job, is simply not a concern at the underlay level because the underlay is a pure Layer 3 routed network.

The underlay routing protocol of choice for most modern leaf-spine builds is eBGP, following the guidance in RFC 7938, which makes the case that BGP is better suited to large-scale datacenter fabrics than any IGP. Each leaf is typically its own autonomous system, the spines are in their own AS, and the BGP sessions between them carry only the loopback addresses of each device, which are what VXLAN tunnels will later terminate on. The result is a simple, flat, highly scalable IP fabric where every device can reach every other device’s loopback via ECMP across multiple spine paths, and adding a new leaf is a matter of cabling it to the spines and bringing up two BGP sessions.

VXLAN: getting Layer 2 to cross a Layer 3 fabric

The underlay being a routed Layer 3 network solves the spanning-tree problem, but it creates a new one: workloads often need Layer 2 adjacency. Virtual machines need to move between physical hosts without changing their IP addresses. Application clusters need to communicate within the same subnet across different leaves. A pure Layer 3 underlay breaks all of that because Layer 2 domains are confined to individual leaf switches.

VXLAN, defined in RFC 7348, solves this by tunneling Ethernet frames inside UDP packets. The idea is that a device at the edge of the fabric, called a VXLAN Tunnel Endpoint or VTEP, takes a regular Ethernet frame from a locally attached server, wraps it in a VXLAN header, an outer UDP header, an outer IP header, and an outer Ethernet header, and sends it across the Layer 3 underlay to the VTEP on the other side. The destination VTEP unwraps the whole thing and delivers the original Ethernet frame to the destination server as if it had traversed a normal Layer 2 segment. From the servers’ perspective, nothing unusual happened. From the network’s perspective, it was just another UDP packet.

The VXLAN header includes a 24-bit VXLAN Network Identifier, or VNI, which is the overlay equivalent of a VLAN ID. Where a VLAN tag gives you 4,094 usable identifiers, a VNI gives you roughly 16 million, which is one of the practical reasons VXLAN took over from older overlay technologies. Each Layer 2 segment in the overlay gets its own VNI, and VTEPs use the VNI to determine which local interface to deliver a decapsulated frame to. In a leaf-spine fabric, the VTEPs are typically the leaf switches themselves, since they’re the devices with direct connections to servers.

VXLAN by itself, though, has a problem. Something has to tell each VTEP where every remote MAC address lives so it knows which destination VTEP to encapsulate toward. Without that information, a VTEP has no choice but to flood unknown traffic to all remote VTEPs, which is exactly the broadcast behavior that made large Layer 2 domains painful in the first place. Early VXLAN deployments dealt with this via multicast groups in the underlay, one multicast group per VNI, so that BUM traffic (Broadcast, Unknown unicast, and Multicast) would reach all VTEPs participating in that VNI. It worked, but it required multicast to be running in the underlay and led to a lot of unnecessary replication. There had to be a better control plane.

EVPN: teaching BGP to distribute MAC addresses

BGP EVPN, standardized in RFC 7432 and extended for VXLAN overlay use in RFC 8365, is that better control plane. The core idea is to run a BGP address family, the L2VPN EVPN address family, between all the VTEPs in the fabric, and use it to distribute MAC and IP reachability information the same way regular BGP distributes IP prefixes.

When a server sends traffic and a leaf switch learns its MAC address locally, instead of just adding that MAC to a local table, the leaf advertises it via EVPN to all other leaves. It does this using a Type-2 route, which carries both the MAC address and the IP address of the host if known, along with the VNI that host belongs to and the IP address of the advertising VTEP. Every other leaf in the fabric receives that advertisement, installs the MAC-to-VTEP mapping, and from that point forward knows exactly where to VXLAN-encapsulate traffic destined for that host. There is no flooding, no multicast group membership, and no runtime MAC learning across the fabric. The control plane handled it.

EVPN also defines a Type-3 route, called an Inclusive Multicast Ethernet Tag route, which VTEPs use to signal that they have a locally attached host in a given VNI. The practical effect is that a VTEP receiving Type-3 routes knows which remote VTEPs participate in each VNI and can build a replication list for BUM traffic, sending a unicast copy to each participating VTEP rather than relying on multicast in the underlay. This is called ingress replication, and it’s the approach most deployments use today because it requires nothing from the underlay beyond basic unicast routing.

Type-5 routes carry IP prefixes rather than individual MAC-IP pairs, and they’re how inter-VNI routing information gets distributed across the fabric when you move to a routed overlay model. More on that in a moment.

The BGP peering topology for EVPN in a leaf-spine fabric typically has the spine switches acting as BGP route reflectors for the leaves. Each leaf peers only with the spines, the spines reflect EVPN routes between leaves, and the result is that every leaf has a complete view of MAC and IP reachability across the entire fabric without a full mesh of peering sessions between all the leaves. It’s the same route reflector pattern that makes iBGP scalable in service provider cores, applied here to the datacenter overlay.

IRB: how routing between VNIs actually works

Getting Layer 2 extension working across the fabric is one thing. Getting Layer 3 routing between different VNIs, which is what lets servers on different subnets communicate, is where Integrated Routing and Bridging comes in, and it’s where the design choices start to have real operational implications.

The simpler model is asymmetric IRB. In this model, the ingress leaf does all the work: it bridges the frame into the correct VNI, then routes it to the destination VNI, then bridges it locally into the outgoing segment. The egress leaf just does a final Layer 2 lookup and delivers the frame. The reason it’s called asymmetric is that the ingress and egress leaves do different amounts of work. The catch is that for this to function, every leaf must have every VNI instantiated locally, even for subnets with no locally attached hosts. As the number of VNIs grows, so does the MAC table footprint on every leaf in the fabric, because every leaf needs to know the MAC addresses for every subnet. This doesn’t scale well once you get to a meaningful number of tenants or subnets.

Symmetric IRB solves the scaling problem, and it’s what most production deployments use. In this model, the ingress leaf routes the frame into a special Layer 3 VNI, which represents a VRF rather than an individual subnet. That Layer 3 VNI is what travels across the VXLAN fabric to the egress leaf. The egress leaf then routes from the Layer 3 VNI into the correct Layer 2 VNI for the destination subnet and bridges the frame locally to the host. Both leaves do the same amount of work in each direction, hence symmetric, and critically, each leaf only needs to instantiate the VNIs for subnets it actually has hosts in. The Layer 3 VNI is the common currency between them. Type-5 EVPN routes carry IP prefix reachability across the fabric so each leaf knows where to route toward subnets it doesn’t host locally.

The anycast gateway capability that EVPN-VXLAN enables is worth calling out specifically. In a traditional routed datacenter, each VLAN has a single gateway router, and that router is a point of concentration for all inter-subnet traffic. In an EVPN-VXLAN fabric, every leaf can be configured as a gateway for every subnet it hosts, all using the same virtual IP and virtual MAC address. A host sends its default gateway traffic to that address, the locally attached leaf handles the routing without the packet having to traverse the fabric to reach a centralized gateway, and the host never knows it’s talking to a distributed routing system. Traffic stays local when it can, which reduces east-west fabric traffic and removes the performance bottleneck of a single gateway device.

What this looks like in practice

In a working production fabric, the full flow for a packet going from one server to another on a different subnet looks something like this. The source server sends an ARP for its default gateway. The locally attached leaf responds immediately with the anycast gateway MAC, because it’s configured as the gateway. The server sends its packet to that MAC. The leaf performs a Layer 3 lookup in the appropriate VRF, finds an EVPN Type-5 route pointing to the remote leaf where the destination prefix lives, encapsulates the packet in VXLAN with the Layer 3 VNI for that VRF, and sends it across the underlay toward the remote VTEP’s loopback address. The remote leaf decapsulates it, does a local Layer 3 lookup to find the destination host’s Layer 2 VNI, does a MAC lookup in that VNI to find the host’s MAC from the previously received Type-2 EVPN route, and delivers the Ethernet frame to the destination server. The whole thing happens without any flooding, without any multicast in the underlay, and without any centralized gateway touching the packet.

That same EVPN control plane also handles multi-homing. If a server connects to two leaf switches for redundancy, EVPN uses an Ethernet Segment Identifier to coordinate between those leaves, ensuring that only one of them advertises reachability for the server at a time in normal operation, and that traffic fails over cleanly if one leaf loses the server connection. This is what replaces the old vPC and MLAG hacks for server-side redundancy, and it does it in a way that’s standardized across vendors rather than locked to a proprietary mechanism.

Where this stands today

EVPN-VXLAN is not a new technology at this point. It’s been in production at scale since the early 2010s at hyperscalers, and the enterprise adoption curve has been steadily climbing ever since the major vendors, Cisco with NX-OS and ACI, Arista with EOS, Juniper with Junos, and Nokia with SR Linux, all converged on interoperable implementations of the same standards. The fact that it’s built on open RFCs rather than proprietary protocols means a multi-vendor fabric is genuinely achievable, though anyone who has tried it will tell you that vendor differences in route-target auto-derivation, ESI handling, and BUM replication behavior still require careful lab validation before you trust it in production.

What’s pushing adoption forward now more than anything is AI infrastructure. The GPU cluster buildouts happening across hyperscalers and enterprises alike have unusually demanding fabric requirements: all-to-all communication patterns at extremely high bandwidth, very low and consistent latency, and support for RoCEv2 (RDMA over Converged Ethernet) which requires a lossless fabric with priority flow control. EVPN-VXLAN fabrics built on leaf-spine Clos topologies with appropriate QoS configuration check all of those boxes, and the combination of VXLAN encapsulation with explicit DSCP markings in the outer header makes it possible to preserve traffic class markings end-to-end across the fabric. It’s one of the reasons the same fabric architecture that was originally designed for VM mobility and multi-tenancy ended up being the right answer for GPU clusters too.

Cisco’s DCNM platform reaching end-of-support in April 2026 has also accelerated migrations, pushing enterprises that had been running legacy VLAN-based fabrics under DCNM management to make a decision. The replacement, Nexus Dashboard Fabric Controller, is built around managing VXLAN EVPN fabrics natively, which has nudged a lot of shops that were on the fence into actually moving.

Is the migration worth it

For anyone still running a three-tier fabric with STP doing real work, the honest answer is that the migration is a meaningful project but the end state is genuinely better in every dimension that matters operationally. You get a fabric that doesn’t have blocked ports, doesn’t have spanning-tree instability as a failure mode, scales horizontally by adding leaves without redesigning anything, and gives you multi-tenancy and workload mobility without vendor-proprietary glue holding it together. The operational model of troubleshooting EVPN routes in BGP is more familiar to anyone with routing experience than the equivalent debugging session in a large STP domain.

The learning curve is real. EVPN-VXLAN introduces a layer of abstraction that takes time to internalize, and the combination of underlay BGP, overlay BGP, VNIs, VTEPs, IRB models, and EVPN route types can feel like a lot when you’re coming from a world where a VLAN was just a VLAN. But the concepts layer cleanly once you understand them in order, and the troubleshooting tools, from show bgp l2vpn evpn to NVE peer state to VTEP reachability verification, are consistent and predictable in a way that STP topology debugging never really was.

The modern datacenter fabric has essentially been decided. Leaf-spine Clos with a BGP underlay and EVPN-VXLAN overlay is what every major vendor ships, what every new enterprise datacenter build defaults to, and what hyperscalers have been running at scale for over a decade. The question for most networks isn’t really whether to go this direction. It’s how to get there from whatever you have running today.