If you run multiple L3Outs in the same VRF on Cisco ACI, there's a classification behavior that catches a lot of engineers off guard: external EPG classification is done per-VRF, not per-L3Out. That single fact can cause traffic from one L3Out to get tagged — and policed — as if it came from a completely different one.
I ran into this in the lab and wanted to break it down in plain terms, because the failure mode is sneaky: everything can look and even work fine, right up until it doesn't.
The Lab Topology
Two leaf switches, one VRF, two L3Outs:
- Leaf-101 — OSPF L3Out on interface
eth1/22(192.168.48.1/24), external EPG pcTag 16387 - Leaf-102 — EIGRP L3Out on an SVI (
192.168.47.1/24), external EPG pcTag 16386 - An App EPG (pcTag 49153) sitting on Leaf-101, with an endpoint at
192.168.1.17/24
OSPF is advertising a host route, 192.168.9.6/32. EIGRP is advertising 192.168.9.1/32. On the EIGRP L3Out, the admin has flagged 192.168.9.0/24 as "External Subnet for External EPG" — a common shortcut to catch a whole range of addresses under one external EPG instead of listing every host route.
Here's the setup visually:
 |
What ACI Actually Does With That /24
This is the part that trips people up. In ACI, an external (L3Out) EPG isn't classified by which physical L3Out or interface the packet arrived on — it's classified by a longest-prefix-match (LPM) lookup against a single table shared by the whole VRF. Every leaf switch running that VRF gets a copy of the same prefix-to-pcTag table.
So the moment 192.168.9.0/24 gets programmed as "external subnet for external EPG" under the EIGRP external EPG, that entry doesn't just live on Leaf-102. It gets pushed to every leaf in the VRF, including Leaf-101 — the one that only knows about OSPF.
Where It Goes Wrong
Before any contract touches the EIGRP external EPG, Leaf-101 only has a generic 0.0.0.0/0 catch-all entry in its prefix table. Traffic from 192.168.9.6 (the OSPF route) matches that default and gets classified correctly as OSPF's external EPG (16387). No drama.
Add just one contract anywhere that references the EIGRP external EPG — even a contract to something entirely unrelated to Leaf-101 — and ACI pushes the 192.168.9.0/24 entry into Leaf-101's classification table too.
Now watch what happens to traffic sourced from 192.168.9.6 hitting Leaf-101 via the OSPF L3Out:
192.168.9.6matches both the OSPF default route (0.0.0.0/0) and the new EIGRP entry (192.168.9.0/24)- LPM always wins with the more specific prefix
/24beats/0, so the packet gets tagged with pcTag 16386 (EIGRP) — not 16387 (OSPF), even though it arrived over the OSPF L3Out and has nothing to do with EIGRP
The switch isn't misbehaving. It's doing exactly what LPM tables are supposed to do. The problem is that two unrelated EPGs ended up sharing address space in a VRF-wide table.
Why It Doesn't Always Blow Up Immediately
In my test, a ping from 192.168.9.6 to the App EPG endpoint still succeeded — even with no contract explicitly permitting OSPF-to-App traffic. Why? Because the traffic got misclassified as EIGRP (16386), and a contract already existed between the EIGRP external EPG and the App EPG. Wrong tag, right permission — so it slipped through.
That's the dangerous part: the traffic looks fine on the surface, but it's being permitted for the wrong reason. And the only reason it worked at all in this lab comes down to one coincidence — which is worth unpacking on its own.
Scenario 2: Add a Third Leaf and the "Lucky Pass" Disappears
The reason the misclassified ping went through in the two-leaf lab is that the App EPG endpoint happened to live on the same leaf (Leaf-101) where the misclassification occurred. ACI doesn't blast every zoning-rule to every leaf in the fabric — a rule only gets programmed onto the leaves where it's actually needed, based on where the relevant EPGs have local significance (an endpoint, a BD, or an L3Out). Leaf-101 already needed the pcTag 16386 ↔ 49153 rule locally, since it happened to host the App EPG endpoint too — so the misdirected OSPF traffic rode along on a rule that was only there for an unrelated reason.
Now change one thing: put the App EPG endpoint on a separate, third leaf (Leaf-103) instead of Leaf-101. The traffic path looks like this:
Here's what happens:
- Traffic from
192.168.9.6still enters through Leaf-101 (the OSPF border leaf) and still gets misclassified as pcTag 16386, for the exact same LPM reason as before. - Policy enforcement in this VRF happens on ingress, so the permit/deny decision for that traffic is made right there on Leaf-101 — before it ever reaches Leaf-103.
- The zoning-rule for pcTag 16386 ↔ 49153 exists on Leaf-103 (which hosts the App EPG endpoint) and on Leaf-102 (which owns the EIGRP L3Out and enforces policy for return traffic heading back out). Neither of those is Leaf-101.
- Leaf-101 has no local reason to carry that specific rule, so the misclassified traffic hits the implicit deny at the ingress leaf and is silently dropped — never even reaching Leaf-103.
So the exact same misclassification bug produces two different outcomes purely based on physical placement of the endpoint: a false "it works" when the endpoint and the misclassifying leaf are the same box, and a silent drop once you scale out to more leaves — which is the normal, realistic case in production. That's what makes this bug so easy to miss in a small lab and so painful to chase down later in a real fabric.
How to Actually Confirm This Is Happening
Three places to check on the leaf CLI:
show zoning-rule scope <vrf-vnid>— shows the actual permit/deny rules per pcTag pair (ACI's ACL equivalent).show system internal policy-mgr prefix— shows what prefixes are currently mapped to which pcTag on that specific leaf. This is where you'll catch the192.168.9.0/24 → 16386entry showing up on a leaf that shouldn't care about EIGRP at all.- ELAM (Embedded Logic Analyzer Module) — trigger a capture on the source IP and check the
sclassfield in the report. If it shows the EIGRP pcTag for OSPF-sourced traffic, that's your smoking gun.
The Takeaway
Don't let external subnets marked "External Subnet for External EPG" overlap across L3Outs that share a VRF. Even if the L3Outs are on completely different leaves, different routing protocols, and different interfaces, the classification table doesn't care — it's VRF-wide.
A few practical guardrails:
- Keep external subnet ranges assigned per L3Out non-overlapping wherever possible.
- Avoid broad "catch-all" subnets (like a /24) on an external EPG unless you're certain no other L3Out in the same VRF will ever advertise something inside that range.
- Before adding a new contract involving an external EPG, check
show system internal policy-mgr prefixon the other leaves in the VRF to see what it's about to push fabric-wide. - If a ping or flow "just works," don't assume it worked through the contract you think it did — verify the pcTag with ELAM if something feels off.
- Don't trust a passing test in a small lab as proof the design is safe — a "lucky pass" caused by endpoint placement on a single leaf can mask a bug that fails as soon as you scale to more leaves.
Related Reading
No comments:
Post a Comment