I am a network professional with over 18 years of experience in enterprise and data‑center networking. I am a CCIE Data Center certified engineer with strong hands‑on expertise in Cisco Nexus and Cisco ACI design, deployment, troubleshooting, and operations. I work on production ACI fabrics and am available for Cisco ACI and Nexus freelancing or consulting work.
Most Nexus-to-ACI migration checklists cover VLANs, contracts, EPGs, and L3Outs in detail. Almost none of them cover the one setting that decides whether your load balancer or firewall HA pair fails over in seconds or in minutes: the Bridge Domain.
This post walks through a real production incident — a ~7-minute application outage caused by a NetScaler HA failover on Cisco ACI — and the two Bridge Domain-level mechanisms responsible for it. Neither mechanism exists on legacy Nexus. Both are silent until the day your HA pair actually fails over.
Table of Contents
- Why ACI Learns Endpoints Differently — And Why That's the Point
- The Incident: A 7-Minute Blackhole That Shouldn't Have Happened
- Root Cause 1 — Rogue Endpoint Soaking
- Root Cause 2 — IP Data-Plane Learning Bounce
- The Fixes: Three Bridge Domain Options, Ranked by Risk
- Migration Checklist: Bridge Domain Settings for Every HA Appliance
- FAQ
1. Why ACI Learns Endpoints Differently — And Why That's the Point {#why-aci-learns-differently}
On a traditional Nexus / legacy switched network, endpoint location is mostly a MAC-address-table problem: a switch sees a source MAC on a port, it learns it, and it moves on. There's no fabric-wide brain keeping track of where every endpoint currently lives — each switch just knows what it personally saw.
Cisco ACI works differently on purpose. Because ACI is Application Centric, not Network Centric, the fabric maintains a live, fabric-wide endpoint database (visible in the APIC as the Endpoint Tracker) that maps every MAC and IP to a specific leaf, port, and encapsulation — updated continuously through both the control plane (COOP, spine proxy) and the data plane (actual traffic flows). Policy, contracts, and forwarding all depend on the fabric knowing precisely where an endpoint is right now, not where it was five minutes ago.
That fabric-wide awareness is exactly what makes features like anywhere-in-the-fabric mobility, consistent policy enforcement, and centralized visibility possible. But it also means ACI has to make judgment calls that a legacy switch never had to make — most importantly, deciding whether a MAC or IP that suddenly "moved" is a legitimate endpoint move, or a loop / rogue device / spoofing attempt.
That judgment call is where our incident began.
2. The Incident: A 7-Minute Blackhole That Shouldn't Have Happened {#the-incident}
The setup: a NetScaler HA pair, primary and secondary, each connected to different port-channels, freshly migrated from legacy Nexus to a Cisco ACI fabric. On the old Nexus network, an HA failover was uneventful — the secondary sends a GARP, MAC tables update, and traffic resumes in seconds.
The same failover on ACI produced a ~7-minute traffic blackhole for the VIP.
The server/NetScaler team had done everything right. The GARP was sent correctly, on time, exactly as designed. The problem was entirely on the ACI fabric side — and it came down to two mechanisms that simply don't exist in a legacy Nexus environment.
3. Root Cause 1 — Rogue Endpoint Soaking {#root-cause-1}
When the NetScaler VIP's MAC address moved from the primary's port-channel to the secondary's port-channel, ACI didn't treat this as a routine endpoint move. It flagged it as a potential rogue endpoint — the same detection logic designed to catch loops, spoofed MACs, or misconfigured devices flapping between ports.
Once flagged, the endpoint entered a soaking period. With the default Rogue EP Detection settings — a 60-second interval and a multiplier of 6 — that soaking period ran for up to 360 seconds before the fabric would accept the MAC at its new location. Fault logs from the incident confirmed three separate soaking cycles before the endpoint was finally relearned correctly.
GARP detection was enabled on the Bridge Domain the entire time — and it made no difference, because GARP detection addresses IP-to-MAC binding updates via ARP, not a MAC's physical relearning across interfaces. That distinction matters and is easy to miss during migration planning.
This entire mechanism — Rogue EP Detection and its soaking timer — does not exist on Nexus. A cross-interface MAC move on Nexus is accepted instantly, with no equivalent quarantine logic.
4. Root Cause 2 — IP Data-Plane Learning Bounce {#root-cause-2}
The second mechanism is more subtle, and arguably more dangerous, because most engineers assume ACI learns IP addresses the same way it learns them on Nexus: from ARP.
It doesn't — at least not by default.
Cisco ACI learns MAC-to-IP bindings from ordinary data-plane traffic, not just from ARP or GARP. During the failover window, the old primary NetScaler node — mid-failover, before it had fully relinquished the VIP — sent TCP RST packets back toward the fabric. Those RSTs carried the VIP's IP address, sourced from the primary's interface. ACI's IP data-plane learning saw that traffic and dutifully relearned the VIP's IP binding back onto the old, now-incorrect primary location.
A single stray TCP RST was enough to bounce the endpoint back to the wrong node, undoing the failover the fabric had just started to converge on.
On legacy Nexus, this scenario doesn't occur, because L3 routing there only updates IP-to-MAC bindings from ARP — a stray data packet from the wrong source has no ability to rewrite the routing table's understanding of where an IP lives.
5. The Fixes: Three Bridge Domain Options, Ranked by Risk {#the-fixes}
All three of these are Bridge Domain-level (or BD + fabric-wide) settings in the APIC. None of them are exotic — they're just easy to leave at default because most migration runbooks never mention them.
✅ Fix 1 — Disable IP Data-Plane Learning on the Bridge Domain Forces ACI to learn IP-to-MAC bindings from ARP/GARP only, matching how legacy Nexus behaves. This directly removes the TCP RST bounce scenario. It's a Bridge Domain policy change and does not require a maintenance window in most environments — verify against your own change process, but this is the lowest-risk of the three fixes.
✅ Fix 2 — Reduce the Rogue EP Detection Multiplier (6 → 2) This cuts the maximum soaking period from roughly 7 minutes down to roughly 2 minutes, while still keeping rogue/loop protection active. It's a tuning change, not a feature removal, and — like Fix 1 — generally doesn't require a maintenance window.
⚠️ Fix 3 — Disable Rogue EP Detection Entirely This gets failover convergence down to seconds, matching legacy Nexus behavior almost exactly. But it also removes a real fabric protection mechanism against MAC spoofing and loop conditions. Treat this as a security-relevant change — it needs a proper risk conversation and sign-off, not a quiet Friday-afternoon toggle.
For most environments, Fix 1 plus Fix 2 gets you 90%+ of the benefit with none of the security trade-off of Fix 3.
6. Migration Checklist: Bridge Domain Settings for Every HA Appliance {#migration-checklist}
If you're moving from Nexus to ACI, this is the review to run before go-live, not after the first outage:
- Identify every appliance that moves a MAC across interfaces during failover — NetScaler, F5, firewalls (active/standby or active/active), load balancers, clustered appliances.
- For each one, check the Bridge Domain it sits on for: IP Data-Plane Learning (enabled/disabled), Rogue EP Detection (multiplier/interval), and whether GARP-Based Detection is actually the right tool for the failover mechanism the appliance uses.
- Don't assume GARP detection alone covers cross-interface MAC moves — it doesn't.
- Schedule and actually execute a controlled failover test for each HA pair, and time the blackhole. Don't extrapolate from the Nexus-era runbook; the convergence numbers are not portable across platforms.
- Document the chosen BD settings per appliance type as part of the migration record, so the next engineer doesn't have to rediscover this the hard way.
Test your failover. Before your users do it for you.
FAQ {#faq}
Q: Does this affect only NetScaler, or other load balancers and firewalls too? Any appliance that moves a MAC address across a different interface or port-channel during a failover event is potentially affected — this includes F5 HA pairs, firewall active/standby clusters, and any clustered appliance using GARP-based failover.
Q: Will disabling IP Data-Plane Learning break anything else on the Bridge Domain? It changes how ACI learns IP-to-MAC bindings fabric-wide for that BD, reverting to ARP/GARP-only learning — the same behavior Nexus has always used. Review it against any endpoints on that BD that rely on being learned purely from data-plane traffic (uncommon, but worth checking) before rolling it out broadly.
Q: Is Rogue EP Detection a security feature I should be reluctant to touch? Yes — treat the soaking multiplier reduction (Fix 2) as a low-risk tuning change, but treat fully disabling Rogue EP Detection (Fix 3) as a security-relevant decision that deserves sign-off, since it removes a real anti-loop/anti-spoofing control.
Q: Why didn't this show up in pre-migration testing? Because most cutover testing validates that traffic passes under normal conditions — not that an HA failover event, specifically, converges within an acceptable time. The gap only surfaces under an actual failover, in production, which is exactly why it needs to be tested deliberately rather than assumed.
Related Articles
If you're working through a Nexus-to-ACI migration, these go deeper on the Bridge Domain and endpoint-learning building blocks referenced above:
- Cisco ACI Static EPG Configuration – Step-by-Step Deployment Guide
- Cisco ACI – Port Channel (eth1/4 & eth1/5) Trunk Configuration for VLAN 420
- Cisco ACI Mis-Cabling Protocol (MCP) – Loop Detection Simplified
- L3Out Subnet Scope Options in Cisco ACI
- COOP: Council of Oracle Protocol - Cisco ACI
- Cisco ACI Service Graph Management Models Explained
Migrating from Nexus to ACI isn't just a config lift-and-shift — it's a change in how the network thinks about where endpoints live. Bridge Domain settings like IP Data-Plane Learning and Rogue EP Detection are exactly the kind of default that looks harmless until the one day it isn't. Review them for every HA appliance before go-live, not after.
No comments:
Post a Comment