Showing posts with label Cisco Nexus. Show all posts

Friday, 31 July 2026

BGP Timers on a VRF in ACI Multi-Site — Why NDO Can't Do It and What Cisco TAC Confirmed

If you've gone looking for a VRF-level BGP Timer Policy in Nexus Dashboard Orchestrator and come up empty, you're not missing a menu — it genuinely isn't there yet. Here's what we found while troubleshooting this on a live Multi-Site fabric with a mix of stretched and local VRFs, the workaround Cisco TAC confirmed as supported, and exactly how to verify the new timers actually took effect instead of just assuming they did.

The gap: NDO does not expose a BGP Timer Policy at the VRF level for ACI Multi-Site — only at the L3Out node level (via L3Out Node Routing Policy → BGP Node Settings).
The workaround: Create and apply the BGP Timer Policy directly on APIC, per site, against the target VRF (local or stretched). Cisco TAC confirmed this is the correct and supported method.
The catch: BGP timers are negotiated during the initial handshake. Changing the policy alone does nothing to an already-established session — you need a hard reset of the BGP session for the new keepalive/hold values to take effect.
Don't trust the config alone. "Configured" timers and "negotiated" (active) timers are two different things in show bgp output — always verify the negotiated value after the reset, not just the configuration.

The Problem: No VRF-Level BGP Timer Option in NDO

In a single-site APIC deployment, applying a custom BGP Timer Policy to a VRF is straightforward — it's a native object under Tenant → Networking → VRF → BGP Context Per Address Family. In Multi-Site environments managed through NDO, that granularity isn't there. NDO's L3Out configuration model lets you attach a BGP Node Routing Policy (with its own keepalive/hold timers) to a node group inside an L3Out template — but there's no equivalent construct for applying timers at the VRF level, independent of a specific L3Out's node groups.

For environments running mixed local and stretched VRFs across sites, that's a real gap: you may want consistent BGP timer behavior tied to the VRF itself, not scattered across every L3Out node group that happens to reference it.

What We Asked Cisco TAC

Rather than guess, we opened a case and asked directly. Here's the exchange, included for anyone hitting the same wall:

Our question:

We need to deploy BGP timers on a VRF within our ACI Multi-Site fabric. Our environment contains both stretched and local VRFs. We do not see an option in NDO to apply BGP timers at the VRF level. Therefore, we are planning to use the following approach: 1) Create a BGP Timer Policy on the APIC. 2) Apply the BGP Timer Policy to the target VRF (local or stretched) via APIC only. Could you please confirm whether the above procedure is the correct and supported method for configuring BGP timers in an ACI Multi-Site environment? Additionally, after applying the policy, we do not see the updated timer values reflected on the BGP neighbour sessions. Is a BGP session reset (hard reset) required for the new timer values to take effect, or are there any additional steps that need to be performed?

Cisco's response:

Configuration Method: Your proposed approach is correct. As NDO does not currently support configuring BGP Timer Policies at the VRF level, applying these policies directly via the APIC on each respective site is the standard and supported method.

Session Reset: Yes, a BGP session reset (hard reset) is required for the new timer values to take effect. Because these timers are negotiated during the initial BGP handshake, the existing sessions must be torn down and re-established to renegotiate the new keepalive and hold-down values with the neighbor.

Two things worth underlining from that response: this is a per-site, APIC-side configuration — NDO is not the tool for VRF-level BGP timers today — and a hard reset is mandatory, not optional, for the values to actually apply.

Step-by-Step: Applying the BGP Timer Policy to a VRF via APIC

Log into the APIC GUI for the site containing the target VRF (do this per site for stretched VRFs — there's no single-pane push for this from NDO).
Navigate to Tenants → <tenant> → Networking → VRFs → <vrf-name>.
Under the VRF, go to the BGP Context Per Address Family (create one for each address family in use — IPv4 and/or IPv6).
Set the BGP Timers field to a custom BGP Timer Policy, or create a new one:
- Go to Tenants → <tenant> → Policies → Protocol → BGP → BGP Timers, right-click and Create BGP Timers Policy.
- Set Keepalive Interval (sec) and Hold Interval (sec) to your target values (keep the standard 1:3 ratio — e.g., 4/12 or 10/30 — most implementations expect hold time to be roughly 3x keepalive).
Apply the policy to the VRF's BGP Context Per Address Family and submit.
Repeat per site for any stretched VRF — this is not propagated automatically across the Multi-Site domain since it's an APIC-local object, not an NDO-managed one.
Confirm the object landed correctly:

   GET https://<apic>/api/node/mo/uni/tn-<tenant>/ctx-<vrf>/bgpCtxAfP-ipv4-ucast.json

Check for the tnBgpTimersPolName attribute pointing to your new policy.

Why the Timers Won't Show Up Until You Reset the Session

This trips people up every time: BGP keepalive and hold timers are exchanged in the OPEN message during session establishment — they're not a live, continuously-enforced parameter that updates on the fly. Changing the policy updates the configuration, but an already-established session keeps running on whatever timers it originally negotiated until it's torn down and rebuilt.

That means after step 7 above, show bgp will show your configured timers, but the negotiated/active timers on any existing session will still be the old values — until you force a reset.

Verifying the Change Actually Took Effect

Don't just trust the APIC MO or the CLI's "configured" line — verify the negotiated value, which is what's actually governing the session.

1. Check the configured value on the border leaf:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

Look for: Configured hold time is 12, keepalive interval is 4 seconds

2. Check the negotiated (active) value in the same output: Look for: Hold time is X, keepalive interval is Y seconds If X/Y still show the old defaults, the session hasn't renegotiated — move to step 3.

3. Force the hard reset:

clear bgp <afi> unicast <peer-ip> vrf <tenant>:<vrf-name>

A soft clear will not renegotiate timers — timers only get exchanged at session establishment, so this has to be a hard clear.

4. Re-check the negotiated value after the reset:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

The negotiated value will be the lower of what each side offers — if your peer's own BGP timer configuration is set higher (or lower) than yours, that governs the final negotiated result, not your configured value alone.

5. Confirm real-world keepalive cadence:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last read"

Run this two or three times a few seconds apart — "Last read" should reset roughly every N seconds, matching your new keepalive interval.

6. Watch for flapping after the change, especially with aggressive timers like 4/12:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last reset"
show logging logfile | include BGP

A Word of Caution on Aggressive Timers

Tighter timers (like 4-second keepalive / 12-second hold) improve failure-detection speed, but they raise the risk of false-positive flaps on links with variable latency, high CPU load on the peer device, or congested control-plane paths. Before rolling this out broadly across a Multi-Site fabric, test it on a single non-critical VRF/L3Out first and monitor for a few days.

FAQ

Does NDO support VRF-level BGP timers at all? Not currently. The only BGP timer construct exposed in NDO is at the L3Out node group level (L3Out Node Routing Policy → BGP Node Settings). VRF-level BGP Timer Policies must be applied directly on APIC, per Cisco TAC's confirmation.

Will this survive an NDO template push or resync? Since the VRF-level BGP Timers Policy is applied directly on APIC and not modeled in the NDO template, it should not be overwritten by a standard template redeploy — but any workflow that deletes and recreates the VRF from NDO would remove it. Treat it as an out-of-band configuration and document it clearly for your team.

Do I need to reset the session on both sides of the peering, or just my side? Just your side's session needs a hard clear to force renegotiation — BGP will tear down and rebuild the full session, which inherently re-triggers the handshake and timer exchange on both ends.

What's a safe keepalive/hold ratio to use? Cisco and most vendors expect hold time to be roughly 3x keepalive (e.g., 4/12, 10/30, 20/60). Going below that ratio, or setting a very low hold time on a link with real latency or jitter, increases false-positive flap risk.

Is this the same procedure for a stretched VRF across multiple sites? Yes, but it must be repeated per site — this is an APIC-local object, and NDO doesn't propagate it across the Multi-Site domain automatically. For a stretched VRF, log into each site's APIC and apply the same BGP Timer Policy independently.

Need help with Cisco ACI, Nexus, data center networking, or network automation?
I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.
Contact me for consulting, troubleshooting, design reviews, and project support.

Thursday, 23 July 2026

Why BGP Maximum-Paths Wasn't the Problem: Understanding Port-Channel Load Balancing and Traffic Imbalance

Recently, we investigated a case where traffic utilization across port-channel member interfaces was significantly unbalanced. One of the links was carrying the majority of the traffic while the other link remained underutilized. Since the environment was already configured with BGP maximum-paths 8, the initial assumption was that BGP load balancing might not be functioning correctly.

After reviewing the configuration, we confirmed that BGP ECMP was operating as expected and that multiple equal-cost paths were available. This shifted our focus from the routing layer to the port-channel load-balancing mechanism.

Why BGP Was Not the Problem

The configuration already included BGP maximum-paths 8, which allows the router to install and use multiple equal-cost paths. This means the network was capable of leveraging several routes simultaneously, eliminating BGP as the primary suspect.

It is important to understand that BGP ECMP and port-channel load balancing serve different purposes. BGP decides which routing path should be used, while the port-channel hashing algorithm determines which physical member link will carry the traffic.

Even when ECMP is working perfectly, traffic can still become concentrated on a single port-channel member if the hashing algorithm maps large flows to the same interface.

Current Hashing Algorithm Analysis

The port-channel was configured with the src-dst-ip enhanced load-balancing algorithm.

This algorithm uses only the source and destination IP addresses to calculate the hash. If a traffic flow continuously uses the same source and destination IP addresses, all packets belonging to that flow will be forwarded through the same member interface.

For example, a database replication stream between two servers will always generate the same hash result. As a result, the entire flow remains pinned to a single physical link regardless of how much unused bandwidth exists on other members of the port-channel.

This behavior is normal because Cisco port-channel load balancing is flow-based rather than packet-based. The objective is to avoid packet reordering and maintain application performance.

Considering src-dst-mixed-ip-port

To improve traffic distribution, we evaluated changing the load-balancing algorithm from src-dst-ip enhanced to src-dst-mixed-ip-port.

Unlike the current configuration, this method includes both Layer 3 and Layer 4 information in the hash calculation. In addition to source and destination IP addresses, it also considers source and destination TCP or UDP port numbers.

This creates a larger number of unique hash combinations and increases the likelihood that different application sessions between the same endpoints will be distributed across multiple member links.

In environments where users, applications, or servers establish numerous simultaneous connections, this approach often results in significantly improved bandwidth utilization across the port-channel.

Important Considerations Before Making the Change

Changing the load-balancing algorithm is generally considered a non-disruptive operation on most Cisco platforms. However, all existing traffic flows will be immediately re-hashed.

As traffic gets redistributed across available links, there may be a brief period of packet reordering. While this is typically minor and transparent to most applications, implementing the change during a maintenance window or low-utilization period is recommended.

Another important consideration is that even with Layer 4 information included, the load-balancing mechanism remains flow-based. If the imbalance is caused by a single high-bandwidth elephant flow, the entire flow will still be assigned to one member link.

In such cases, changing the hashing algorithm may provide only limited improvement.

Recommended Validation After the Change

After implementing the new load-balancing method, monitor interface utilization and traffic patterns for several hours.

Review the following:

Port-channel member utilization

Top bandwidth-consuming flows

NetFlow or telemetry statistics

Application traffic distribution

Interface counters

If traffic becomes more evenly balanced across the member interfaces, the change has achieved its objective. If the imbalance continues, further investigation should focus on identifying large elephant flows or application-specific traffic patterns.

Conclusion

Based on our analysis, BGP maximum-paths was not contributing to the bandwidth imbalance. The routing layer was functioning correctly and supporting multiple equal-cost paths as designed.

The more likely cause was the src-dst-ip enhanced hashing algorithm, which uses only Layer 3 information and can result in traffic concentration when a small number of large flows dominate bandwidth consumption.

Moving to src-dst-mixed-ip-port is a logical and widely adopted optimization because it introduces Layer 4 awareness into the hashing calculation and generally improves traffic distribution when multiple flows exist between the same source and destination hosts.

While it may not solve scenarios involving a single elephant flow, it represents the most appropriate next step before exploring more advanced traffic-engineering options

Friday, 26 June 2026

Cisco ACI vPC Explained – Architecture, Working, Traffic Flow, Configuration, Best Practices & Interview Questions

Cisco ACI vPC Explained: Architecture, Working, Benefits & Traffic Flow

High availability is one of the most important design goals in modern data centers. Whether you are deploying virtual machines, physical servers, firewalls, or storage arrays, network redundancy is essential to eliminate single points of failure.

Cisco Application Centric Infrastructure (ACI) provides a powerful feature called Virtual Port Channel (vPC) that allows an endpoint to connect simultaneously to two different leaf switches while appearing as a single logical switch from the endpoint's perspective. This design delivers redundancy, active-active forwarding, and efficient bandwidth utilization without relying on traditional Spanning Tree Protocol (STP) blocking.

In this guide, you'll learn:

What Cisco ACI vPC is
Why vPC is required
How Cisco ACI vPC works internally
Differences between traditional Nexus vPC and ACI vPC
MCT architecture
ZMQ communication
Traffic flow
Design options
Best practices

Whether you're preparing for the CCNP Data Center, CCIE Data Center, or working in a production ACI environment, this guide will provide a solid understanding of Cisco ACI vPC.

What is Cisco ACI vPC?
Why Do We Need vPC?
Traditional Network Challenges
Cisco ACI vPC Architecture
Components of vPC
MCT Architecture Explained
How Peer Communication Works
ZMQ and URIB Explained
Traffic Flow in Cisco ACI vPC
Benefits of Cisco ACI vPC
Design Best Practices

What is Cisco ACI vPC?

A Virtual Port Channel (vPC) in Cisco ACI enables two independent leaf switches to present themselves as a single logical switch to a connected device such as:

Physical servers
VMware ESXi hosts
Hyper-V hosts
Firewalls
Load Balancers
Storage Arrays
Traditional Ethernet switches

The connected endpoint forms one LACP Port Channel, but the physical links terminate on two separate ACI leaf switches.

This provides:

✅ Link redundancy

✅ Switch redundancy

✅ Active-active forwarding

✅ Increased bandwidth

✅ Zero blocked links

Unlike traditional Layer 2 designs, both links remain forwarding simultaneously.

Why Do We Need vPC?

Imagine a server connected to only one switch.


Server
   |
 Leaf201

If Leaf201 fails, the server immediately loses connectivity.

Now imagine connecting the server to two switches without vPC.


      Server
      /    \
Leaf201   Leaf202

This creates a Layer-2 loop.

Traditional Ethernet networks solve loops using Spanning Tree Protocol (STP).

Unfortunately STP blocks one of the redundant links, wasting available bandwidth.

ACI vPC eliminates this limitation by allowing both links to remain active.

Result:

No blocked ports
Better utilization
Higher availability
Faster convergence

Traditional Nexus vPC vs Cisco ACI vPC

Many engineers assume ACI vPC works exactly like traditional Cisco Nexus vPC.

It does not.

Traditional Nexus vPC	Cisco ACI vPC
Uses dedicated peer-link	No dedicated peer-link
Uses CFS messaging	Uses ZMQ messaging
Manual synchronization	Fabric-based synchronization
Standalone switches	Fabric-managed leaf switches
Peer keepalive required	Fabric manages peer communication

This architectural difference is one of the biggest reasons Cisco ACI scales much better in large data centers.

Cisco ACI vPC Architecture

A typical deployment looks like this.


             Spine101
                 |
        -------------------
        |                 |
    Leaf201           Leaf202
        \               /
         \             /
          \           /
           Server (LACP)

Both Leaf201 and Leaf202 participate in a vPC domain.

The server believes it is connected to a single logical switch.

Internally, however, both leaf switches coordinate forwarding decisions through the ACI fabric.

Key Components of Cisco ACI vPC

1. Leaf Switches

Leaf switches provide endpoint connectivity.

Each endpoint connects to one or more leaf switches.

For vPC deployments:

Two leaf switches form one logical vPC pair.
Both switches actively forward traffic.
Either switch can independently forward packets to the spine layer.

2. Spine Switches

Spine switches never connect directly to endpoints.

Their responsibilities include:

Forwarding traffic between leaves
Maintaining fabric connectivity
Providing equal-cost paths
Supporting IS-IS routing inside the fabric

Every leaf switch connects to every spine switch.

3. APIC Controller

The Application Policy Infrastructure Controller (APIC) is the management plane of Cisco ACI.

APIC performs:

Policy management
Automation
Monitoring
Fabric discovery
Endpoint learning
Configuration deployment

Importantly, APIC does not forward data traffic.

Even if APIC becomes unavailable, data forwarding continues because forwarding decisions are distributed across the fabric.

4. LACP Port Channel

The endpoint uses IEEE 802.3ad LACP.

Instead of seeing two independent switches, the endpoint sees one logical port channel.

This allows:

Load balancing
Automatic failure detection
Link aggregation
Active-active forwarding

Understanding MCT Architecture

One of the biggest differences between traditional Nexus vPC and Cisco ACI is the implementation of Multichassis Trunking (MCT).

Traditional Nexus switches require a dedicated peer-link between vPC peers.


Leaf1 -------- Peer Link -------- Leaf2

Cisco ACI removes this dependency.

Instead, synchronization occurs through the fabric itself.


Leaf201
    |
 Spine
    |
Leaf202

Benefits include:

Simpler cabling
No dedicated peer-link
Better scalability
Reduced operational complexity

This architecture allows leaf switches to synchronize state information without requiring a separate physical interconnect dedicated to vPC.

How Peer Communication Works

Cisco ACI uses the fabric network to exchange state information between vPC peers.

Internally:

Leaf201 discovers Leaf202 through the ACI fabric.
IS-IS establishes routing information.
URIB learns the peer's reachability.
The vPC Manager receives routing updates.
The vPC Manager establishes a communication channel using ZeroMQ (ZMQ).
Both leaf switches synchronize operational state for the vPC.

If the route to the peer becomes unavailable, the vPC Manager is notified and the logical MCT relationship is taken down accordingly, helping maintain a consistent operational state. This behavior aligns with Cisco's ACI vPC architecture and avoids relying on a dedicated peer-link.

What is ZeroMQ (ZMQ)?

One of the most common interview questions is:

Why does Cisco ACI use ZMQ instead of CFS?

ZeroMQ (ZMQ) is a lightweight, high-performance messaging library that Cisco ACI uses for communication between vPC peer switches.

Instead of sending synchronization data over a dedicated peer-link, the ACI fabric transports these messages over IP connectivity between the leaf switches.

Advantages of ZMQ include:

Faster communication
Lower overhead
High scalability
Reliable message delivery
Better support for large-scale ACI fabrics

This messaging mechanism is one of the reasons Cisco ACI can simplify vPC design compared to traditional NX-OS implementations.

Understanding URIB

URIB (Unicast Routing Information Base) is responsible for maintaining routing information on each leaf switch.

The vPC Manager subscribes to URIB updates.

Whenever a new route to the peer leaf becomes available, URIB notifies the vPC Manager, allowing it to establish the required communication session.

If the route disappears because of a failure, URIB notifies the vPC Manager again so it can update the operational state appropriately.

Benefits of Cisco ACI vPC

Organizations deploy Cisco ACI vPC because it provides:

High Availability: Loss of a single link or leaf switch does not interrupt connectivity.
Active-Active Forwarding: Both uplinks remain in service, maximizing bandwidth utilization.
Simplified Operations: No dedicated peer-link reduces cabling and operational complexity.
Faster Convergence: Failures are detected and handled quickly, minimizing application downtime.
Scalability: Fabric-based synchronization supports large-scale data center deployments.
Efficient Load Balancing: Traffic is distributed across all active links.

Coming Up in Part 2

In the next part, we'll cover:

Cisco ACI vPC Design Options (Combined vs Individual Profiles)
Packet Flow Explained Step by Step
Configuration Workflow in APIC
Common Configuration Mistakes
Best Practices
Troubleshooting Commands
20 Cisco ACI vPC Interview Questions
FAQ Section (Schema-ready)
Conclusion
Related Reading from Your Blog

📚 Related Cisco ACI Articles

If you're learning Cisco ACI from the ground up, these articles will help you understand the technologies that work together with Virtual Port Channel (vPC).

1. Cisco ACI Explained – Concepts, Learning Prerequisites, Benefits & Interview Questions

If you're new to Cisco ACI, start with this comprehensive guide that covers the core architecture, policy model, and key building blocks before diving into advanced topics like vPC. It provides a strong foundation for understanding how the ACI fabric operates. Cisco ACI Explained – Concepts, Learning Prerequisites, Benefits & Interview Questions

2. Understanding VLAN Pool Roles in Cisco ACI

vPC deployments often use VLAN Pools to map VLAN encapsulations for endpoint connectivity. Learn the difference between Internal and External (On-the-Wire) VLAN Pool roles and understand when each should be used in production environments. Understanding VLAN Pool Roles in Cisco ACI

3. Understanding Domain Types in Cisco ACI

Before configuring vPC, it's important to understand Physical Domains, L3 Domains, Fibre Channel Domains, and External Bridge Domains. This article explains where each domain type fits within the ACI policy model. Understanding Domain Types in Cisco ACI

4. Key Concepts of Application Profile in Cisco ACI

Application Profiles organize Endpoint Groups (EPGs) that communicate using policies and contracts. This guide explains how Application Profiles fit into the ACI hierarchy and why they're essential for application-centric networking. Key Concepts of Application Profile in Cisco ACI

5. Cisco ACI Static EPG Configuration – Step-by-Step Guide

After creating a vPC, you'll typically bind servers to an Endpoint Group (EPG). This practical walkthrough demonstrates how to configure a static EPG, associate it with a Bridge Domain, and apply the required policies. Cisco ACI Static EPG Configuration – Step-by-Step Deployment Guide

6. Cisco ACI Port Channel Configuration (eth1/4 & eth1/5)

Want to configure a Port Channel in Cisco ACI? This article provides a detailed step-by-step guide for creating a Port Channel using LACP, configuring interface policies, AAEPs, domains, and deploying a Static EPG. It's an ideal follow-up after understanding vPC concepts. Cisco ACI Port Channel (eth1/4 & eth1/5) Trunk Configuration for VLAN 420

7. Configuring Port Profiles in Cisco ACI

Learn how Port Profiles work in Cisco ACI, including converting uplink ports to downlink ports using NX-OS style CLI. Understanding interface profiles and policy groups will help you design flexible and scalable vPC deployments. Configuring Port Profiles in Cisco ACI

8. L3Out Subnet Scope Options in Cisco ACI

Many production environments use vPC together with L3Out connections. This guide explains the different L3Out subnet scope options, including export, import, shared route control, and security import subnets, helping you design secure external connectivity. L3Out Subnet Scope Options in Cisco ACI

9. What is a Contract Preferred Group in Cisco ACI?

Contract Preferred Groups simplify communication between Endpoint Groups (EPGs) within the same VRF by reducing the need for explicit contracts. Learn when to use this feature and how it affects traffic flow in Cisco ACI. What is a Contract Preferred Group in ACI?

Sunday, 26 April 2026

Cisco ACI “Unknown” Leaf State Explained: Certificates, LLDP, Software, and Hardware Issues

In a Cisco ACI fabric, one of the most frustrating issues during initial fabric bring‑up, expansion, or node replacement is seeing a leaf switch stuck in an “Unknown” state. When a leaf is in an unknown state, it means the APIC cannot fully discover, authenticate, or manage the node, preventing it from joining the fabric and participating in traffic forwarding.

This issue can occur during initial fabric deployment, adding a new leaf to an existing fabric, replacing failed hardware, performing software upgrades, or moving switches between fabrics.

Understanding why a leaf enters the “Unknown” state is critical for fast recovery. In most cases, the root cause is not a single configuration mistake but a failure in communication, authentication, compatibility, or initialization.

This article explains the most common causes of the “Unknown” leaf state in Cisco ACI, why they happen, and how to systematically troubleshoot them in real‑world environments.

1. What Does “Unknown” Leaf State Mean in Cisco ACI?

When a leaf is shown as “Unknown” in the APIC GUI, it indicates that the APIC can see the node attempting discovery, but the node cannot complete secure authentication or critical control‑plane messaging has failed.

At this stage, the leaf is not operational, not programmable, and cannot forward production traffic.

2. Certificate Issues Between Leaf and APIC

Cisco ACI uses mutual certificate‑based authentication between the APIC controllers and fabric nodes. Every leaf switch must present a valid certificate chain that is signed and trusted by the APIC.

If the certificate exchange fails, the leaf cannot authenticate correctly, and APIC marks it as Unknown.

Common certificate‑related problems include an invalid or corrupted certificate on the leaf, the leaf previously belonging to another ACI fabric, expired or mismatched certificates due to time drift, or incomplete cleanup after node replacement.

These issues are often seen when hardware is reused without full re‑initialization.

The most reliable resolution is to completely wipe and reinitialize the leaf switch, ensure it boots in ACI mode, and allow APIC to generate and install a fresh certificate.

3. LLDP Mismatch or LLDP Failure

Cisco ACI relies heavily on LLDP for fabric discovery and adjacency validation. LLDP is mandatory in ACI for identifying correct topological relationships between leaf and spine switches.

If LLDP is not exchanged correctly, discovery fails and the leaf remains in an Unknown state.

Typical LLDP problems include LLDP being disabled on connected devices, LLDP filtered due to security policies, incorrect cabling such as connecting a leaf to something other than a spine, or the switch running in NX‑OS mode instead of ACI mode.

Symptoms include missing neighbor information, partial discovery, or interfaces appearing operationally down.

To resolve LLDP issues, ensure LLDP is enabled end‑to‑end, verify correct cabling from leaf to spine only, confirm the switch is running in ACI mode, and check optics and interfaces on both ends.

4. Firmware or Software Incompatibility

ACI fabric components are designed to work within a compatible software matrix. Significant software mismatches between the APIC, leaf, and spine can prevent successful node onboarding.

This often occurs when a leaf is running an unsupported ACI version, the APIC has been upgraded but the leaf image was not updated, or an incorrect software image is installed on the switch.

Typical symptoms include the leaf being detected but never transitioning from Unknown to Active, along with compatibility or image‑related faults.

Resolution requires verifying Cisco’s supported version matrix and ensuring that the leaf software version is compatible with both the APIC and spine versions.

5. Hardware Problems

Physical layer issues are a common but frequently overlooked cause of Unknown leaf state. Even a simple faulty optic can completely prevent discovery.

Common hardware causes include defective or unsupported transceivers, damaged fiber or copper cables, faulty ports on the leaf or spine, or mismatched speed or media types.

Indicators include interfaces staying down, intermittent connectivity, missing LLDP information, or hardware‑related faults in APIC.

Troubleshooting involves replacing suspect cables and optics, using only Cisco‑supported transceivers, testing alternate ports, and validating interface status on both leaf and spine.

6. Time Synchronization Issues

Certificate validation in ACI is time‑sensitive. If the system time on the leaf is significantly out of sync with the APIC, certificate authentication can fail even if the configuration and connectivity are correct.

This is common in environments where NTP is misconfigured, unavailable, or the device has been powered off for an extended period.

Symptoms include authentication failures and persistent Unknown leaf state with no obvious physical or configuration issues.

Resolution involves verifying NTP configuration on APIC, ensuring the leaf can synchronize time, and reinitiating discovery after time correction.

7. Incorrect Node ID or Serial Number Issues

ACI uniquely identifies nodes using a combination of node ID, serial number, and certificates. If these identifiers do not match what APIC expects, the leaf will fail authentication.

This commonly occurs when a switch was previously part of another fabric, reused after RMA without proper cleanup, or when a node ID conflict exists.

Symptoms include the leaf appearing with unexpected identity information or being rejected during registration.

The safest resolution is to fully wipe the leaf configuration, reboot the device, and allow APIC to assign a fresh node identity.

8. Recommended Troubleshooting Sequence

When a leaf is stuck in Unknown state, follow this sequence:

First, verify physical connectivity and optics.
Second, confirm LLDP adjacency and cabling.
Third, check software compatibility.
Fourth, validate certificates and authentication.
Fifth, ensure correct time synchronization.
Finally, reinitialize the leaf if needed.

Following this order avoids unnecessary configuration changes and reduces downtime.

9. Best Practices to Prevent Unknown Leaf State

Always wipe reused hardware before deployment.
Keep APIC, spine, and leaf software versions compatible.
Use supported Cisco optics and cables.
Ensure stable NTP configuration.
Verify LLDP connectivity during installation.
Document node IDs and serial numbers carefully.

Most Unknown leaf issues are preventable with proper procedures.

10. Conclusion

An Unknown leaf state in Cisco ACI is always a symptom of a failed discovery, authentication, compatibility, or initialization process. Certificate issues, LLDP failures, firmware incompatibility, hardware problems, time synchronization issues, and incorrect node identity are the most common causes.

By understanding these root causes and following a structured troubleshooting approach, engineers can resolve Unknown leaf issues quickly and avoid prolonged deployment delays.

A clean initialization and methodical verification remain the most effective solution in Cisco ACI environments.

Cisco ACI L3Out Interview Questions Explained – Design, Implement, and Troubleshooting

Section 1: Basic Cisco ACI L3Out Interview Questions

1. What is L3Out in Cisco ACI?

L3Out (Layer‑3 Outside) is the ACI construct that provides external Layer‑3 connectivity between the ACI fabric and networks outside the fabric.

2. Why do we need L3Out?

L3Out is used to:

Connect ACI to external routers
Integrate firewalls
Provide north‑south traffic
Advertise routes between ACI and external networks

3. Is L3Out mandatory in ACI?

No. L3Out is required only if the ACI fabric needs external Layer‑3 communication.

4. Where is L3Out configured?

L3Out is configured under a Tenant, associated with a VRF, and deployed on leaf switches.

5. Is L3Out Layer‑2 or Layer‑3?

L3Out is strictly a Layer‑3 construct.

Section 2: L3Out Components Interview Questions

6. What are the main components of L3Out?

L3Out object
Logical Node Profile
Logical Interface Profile
External EPG
Contracts

7. What is a Logical Node Profile?

It defines which leaf nodes participate in the L3Out.

8. What is a Logical Interface Profile?

It defines:

Interface type (routed, SVI)
IP addressing
Encapsulation (VLAN)
Connectivity to external device

9. Can L3Out be deployed on multiple leafs?

Yes. L3Out is commonly deployed on multiple leaf switches for redundancy.

10. What happens if an L3Out leaf fails?

Traffic fails over to other L3Out‑enabled leafs, assuming proper design (ECMP / routing).

Section 3: L3Out and Routing Protocol Interview Questions

11. Which routing protocols are supported with L3Out?

Static routing
OSPF
BGP

12. Which routing protocol is most commonly used?

BGP, due to scalability and flexibility.

13. Is OSPF supported in L3Out?

Yes, but less commonly used in large deployments.

14. Can static routes be used in L3Out?

Yes, for simple or small environments.

15. Can L3Out support ECMP?

Yes. ACI supports ECMP for L3Out when routing protocols allow it.

Section 4: L3Out and VRF Association Questions

16. Is L3Out associated with a VRF?

Yes. Every L3Out must be associated with exactly one VRF.

17. Can one L3Out be shared across multiple VRFs?

No. One L3Out belongs to only one VRF.

18. Can multiple L3Outs exist in the same VRF?

Yes. A VRF can have multiple L3Outs.

19. Why would you create multiple L3Outs in one VRF?

Multiple external devices
Separate routing domains
Different security or routing policies

20. What happens if VRF association is wrong?

External routing will fail and traffic will be dropped.

Section 5: External EPG Interview Questions

21. What is an External EPG?

An External EPG represents external networks outside the ACI fabric.

22. Why is an External EPG required?

Because ACI is deny‑by‑default, and external networks must also follow ACI security policy.

23. How is traffic allowed between internal EPGs and External EPGs?

Using contracts.

24. Is External EPG similar to internal EPG?

Conceptually yes, but it represents external endpoints.

25. Can there be multiple External EPGs under one L3Out?

Yes.

Section 6: L3Out and Contracts (Very Important)

26. Is traffic allowed by default between ACI and external networks?

No. Traffic is denied by default.

27. How do you allow internal traffic to external networks?

Apply contracts between internal EPG and External EPG.

28. Can External EPG be provider or consumer?

It can be either or both, depending on traffic flow.

29. What happens if no contract is applied?

Traffic will be dropped, even though routing is correct.

30. Why do many L3Out issues occur?

Because routing works, but contracts are missing or incorrect.

Section 7: L3Out Design Interview Questions

31. Routed Interface vs SVI – what is preferred?

Routed interfaces are preferred for simplicity and scale.

32. When would you use SVI‑based L3Out?

When connecting to:

Traditional VLAN‑based networks
Legacy firewalls

33. Can L3Out connect to firewalls?

Yes, very commonly.

34. Can one firewall connect to multiple L3Outs?

Yes, depending on design.

35. Should L3Out be deployed on border leafs?

Yes. Border leafs are best practice.

Section 8: Advanced L3Out Interview Questions

36. How is route leaking handled in ACI?

Using Shared Services VRF and contracts.

37. Can L3Out be used with Shared Services VRF?

Yes, very commonly.

38. Can L3Out be stretched across sites?

Multi‑Pod: Yes
Multi‑Site: Via individual site L3Outs

39. How does L3Out behave in Multi‑Pod?

L3Out is shared across pods.

40. How does L3Out behave in Multi‑Site?

Each site has its own L3Out, orchestrated by NDO.

Section 9: L3Out and External Connectivity Troubleshooting Questions

41. Routing is correct but traffic fails – why?

Most likely contract or filter issue.

42. Endpoint can ping gateway but not internet – why?

External EPG contract missing or incorrect.

43. How to verify routes learned from L3Out?

APIC routes view
Leaf show commands
moquery

44. How do you verify contract programming?

Use:

show zoning-rule

45. How do you verify L3Out operational status?

APIC Health score
Faults
Leaf CLI

Section 10: MoQuery Commands for L3Out Verification

46. Verify L3Out configuration

moquery -c l3extOut

47. Verify External EPGs

moquery -c l3extInstP

48. Verify L3Out subnets

moquery -c l3extSubnet

49. Verify VRF association

moquery -c fvCtx

50. Check faults related to L3Out

moquery -c faultInst

Section 11: Common L3Out Mistakes (Interview Favorite)

51. Forgetting contracts

Most common mistake.

52. Wrong VRF association

Causes route blackholing.

53. Deploying L3Out on wrong leaf

Traffic won’t exit properly.

54. Using SVI instead of routed interface unnecessarily

Adds complexity.

55. Not planning for redundancy

Leads to single‑point failures.

Section 12: Scenario‑Based L3Out Interview Questions

56. When should you create multiple External EPGs?

When different external networks need different security policies.

57. Can multiple L3Outs advertise the same prefix?

Yes, but routing behavior must be carefully designed.

58. Can L3Out connect to non‑Cisco devices?

Yes. ACI is vendor‑agnostic at Layer‑3.

59. Can L3Out be used for Internet access?

Yes, with proper NAT/firewall integration.

60. What is the biggest design challenge in L3Out?

Balancing security, simplicity, and scalability.

Conclusion

Cisco ACI L3Out is the gateway between the ACI fabric and the external world. Interviews around L3Out focus on design understanding, security enforcement, VRF association, and troubleshooting approach, not just configuration steps.

If you understand:

How routing works
Why contracts are mandatory
Where L3Out should be placed
How to verify and troubleshoot

you will handle most Cisco ACI L3Out interview questions confidently.

✅ Interview Tip

When answering L3Out questions, always explain:

Routing
Security (contracts)
Placement (leafs)
Verification

Cisco ACI Interview Questions and Answers (ESG, Multi‑Site, NDO, MoQuery Explained)

Cisco Application Centric Infrastructure (ACI) is a cornerstone technology in modern enterprise data centers. As a result, Cisco ACI interview questions appear frequently in interviews for Network Engineers, Data Center Specialists, ACI Architects, and CCIE Data Center candidates.

This comprehensive guide brings together basic, intermediate, and advanced ACI interview questions, including Endpoint Security Groups (ESG), Multi‑Pod, Multi‑Site, and Nexus Dashboard / NDO, with concise, practical answers. It also includes comparison tables frequently used by interviewers to test real‑world understanding.

Section 1: Cisco ACI Fundamentals – Core Interview Questions

1. What is Cisco ACI?
Cisco ACI is a policy‑based data center networking solution that centralizes management and enforces application‑centric policies across a fabric.

2. What problem does Cisco ACI solve?
It reduces operational complexity, configuration drift, and scalability issues found in traditional networking.

3. Which switches are used in ACI?
Cisco Nexus 9000 series switches running in ACI mode.

4. What is APIC?
APIC (Application Policy Infrastructure Controller) is the centralized control and management platform for the ACI fabric.

5. Is APIC part of the data path?
No. APIC is out of the data path; traffic continues even if APIC is unavailable.

Section 2: ACI Architecture Interview Questions

6. What topology does ACI use?
Leaf–spine architecture.

7. What connects to leaf switches?
Endpoints such as servers, firewalls, load balancers, and L3Outs.

8. What is the role of spine switches?
High‑speed packet forwarding between leaf switches.

9. Can endpoints connect to spine switches?
No.

10. What happens if a spine fails?
Traffic reroutes through remaining spines without impact.

Section 3: ACI Logical Model Questions

11. What is a Tenant?
An administrative boundary representing an organization or business unit.

12. What is a VRF in ACI?
A Layer‑3 routing domain providing IP isolation.

13. What is a Bridge Domain (BD)?
A Layer‑2 forwarding domain that defines flooding and gateway behavior.

14. What is an Endpoint Group (EPG)?
A logical group of endpoints that share the same policy.

15. Is an EPG the same as a VLAN?
No. EPGs are policy objects, not VLANs.

Section 4: Traffic Flow and Contracts Interview Questions

16. What is the default traffic behavior in ACI?
Traffic between EPGs is denied by default.

17. How is traffic allowed?
Using contracts.

18. What is a contract?
A policy object that defines who talks, what traffic is allowed, and direction.

19. What are subjects?
Logical groupings of filters within a contract.

20. What is a filter?
Defines protocol, port, and direction.

Section 5: Advanced Policy – vzAny and Taboo

21. What is vzAny?
A special object that represents all EPGs within a VRF.

22. Why use vzAny?
To simplify policy and reduce TCAM usage.

23. What is a Taboo Contract?
A deny contract used to explicitly block traffic.

24. Does Taboo override permit contracts?
Yes. Deny always takes precedence.

25. When should Taboo be used?
Only for specific, unavoidable deny cases.

Section 6: Endpoint Security Group (ESG) Interview Questions

26. What is an ESG?
Endpoint Security Group is a policy‑based security construct independent of topology.

27. How is ESG different from EPG?
EPG is topology‑based; ESG is security‑policy‑based.

28. Can ESG span multiple EPGs?
Yes.

29. Does ESG use contracts?
Yes, contracts are applied directly between ESGs.

30. Is ESG mandatory?
No, it is optional and mainly used for zero‑trust designs.

🔍 Comparison Table: EPG vs ESG

Feature	EPG	ESG
Based on	Topology	Security policy
Dependency	BD / VLAN	Independent
Scope	Limited	Cross‑EPG
Zero‑Trust	Basic	Strong
Use Case	General policy	Advanced security

Section 7: ACI Multi‑Pod Interview Questions

31. What is ACI Multi‑Pod?
A single ACI fabric stretched across multiple locations (pods).

32. Is Multi‑Pod one fabric?
Yes.

33. How many APICs manage Multi‑Pod?
One APIC cluster.

34. Are L2 and L3 stretched?
Yes.

35. What is IPN?
Inter‑Pod Network connecting pods.

36. What is the main risk of Multi‑Pod?
Increased fault domain.

Section 8: ACI Multi‑Site Interview Questions

37. What is ACI Multi‑Site?
Multiple independent ACI fabrics managed under common policy.

38. Are fabrics independent?
Yes.

39. Is Layer‑2 stretched in Multi‑Site?
No, Multi‑Site is primarily Layer‑3.

40. What is Multi‑Site mainly used for?
Disaster recovery and geo‑redundancy.

🔍 Comparison Table: Multi‑Pod vs Multi‑Site

Feature	Multi‑Pod	Multi‑Site
Fabric	Single	Multiple
APIC	Shared	Separate
L2 Stretch	Yes	No
Latency Requirement	Strict	Relaxed
Fault Isolation	Low	High
Best Use Case	Metro DC	Geo‑redundancy

Section 9: Nexus Dashboard & NDO Interview Questions

41. What is Nexus Dashboard (ND)?
A unified platform hosting ACI‑related services like NDO, NDI, and Insights.

42. What is Nexus Dashboard Orchestrator (NDO)?
A tool used to orchestrate policies across multiple ACI sites.

43. What was NDO previously called?
MSO (Multi‑Site Orchestrator).

44. Does NDO replace APIC?
No.

45. What is a schema in NDO?
A logical template defining tenants and policies.

Section 10: Nexus Dashboard Insights (NDI)

46. What is NDI?
Nexus Dashboard Insights provides health analytics, anomaly detection, and assurance.

47. Does NDI configure the fabric?
No. It is analytics only.

48. Is NDI mandatory?
No, but highly recommended for large environments.

🔍 Comparison Table: ND vs NDO vs NDI

Component	Purpose
Nexus Dashboard	Platform
NDO	Policy orchestration
NDI	Analytics & assurance
APIC	Fabric control

Section 11: Troubleshooting Interview Questions

49. What is a health score?
A numeric representation of object health.

50. What is a fault?
An abnormal condition detected in the fabric.

51. What is moquery?
A read‑only CLI tool to query ACI managed objects.

52. Is moquery safe in production?
Yes.

53. Why prefer moquery over GUI?
Speed and accuracy.

Section 12: Automation and Operations

54. Does ACI support automation?
Yes, via native REST APIs.

55. Can Ansible be used with ACI?
Yes.

56. What is Day‑0?
Fabric deployment.

57. What is Day‑1?
Policy configuration.

58. What is Day‑2?
Operations and troubleshooting.

Section 13: ACI Disadvantages (Interview Favorite)

59. What is the biggest challenge in ACI?
Learning curve.

60. Is ACI expensive?
Yes, compared to traditional designs.

61. Is ACI vendor locked?
Yes.

62. Can ACI be over‑engineered?
Yes, with poor design.

Conclusion

Cisco ACI interviews test more than definitions—they assess design thinking, security understanding, architecture choice, and operational awareness. A clear grasp of EPG vs ESG, Multi‑Pod vs Multi‑Site, and NDO vs NDI is critical for senior‑level roles.

If you understand why ACI behaves the way it does, not just how to configure it, you will stand out in interviews.

Sunday, 7 September 2025

How Rogue Endpoint Detection Works in Cisco ACI

Cisco ACI continuously monitors endpoint behavior and identifies abnormal movement patterns.

Key Actions Performed:

Detects endpoints moving frequently across leaf switches
Marks the endpoint as rogue
Converts the endpoint entry into a static entry
Deletes the endpoint after a configured timeout
Generates faults and alerts for visibility
Sends host tracking packets to relearn correct location

👉 This ensures stability while preventing network disruption.

🔄 Behavior Based on Cisco ACI Version

Before Version 3.2(6)

Endpoint is marked as static
Traffic is dropped during quarantine
MAC/IP entry is deleted after timeout

👉 Impact:
This behavior was highly disruptive because legitimate traffic could be blocked.

Version 3.2(6) and Later

Endpoint is marked as static
Traffic is allowed even during quarantine
MAC/IP entry is deleted after timeout

👉 Improvement:
From version 3.2(6), Cisco improved the design to ensure:

Minimal traffic disruption
Better user experience
Continued monitoring of rogue behavior

📊 Quick Comparison

Feature	Before 3.2(6)	After 3.2(6)
Endpoint Handling	Static	Static
Traffic During Quarantine	Dropped	Allowed
Network Impact	High	Low
Stability	Moderate	High

📝 Rogue / COOP Exception List

✅ Why It Is Needed

Some endpoints (like load balancers, clustered systems, or hypervisors) may naturally move frequently and should not be flagged as rogue.

📋 How Exception List Works

Allows higher tolerance for endpoint movement
Endpoint is marked rogue only after 3000 moves in 10 minutes
Once marked:
- Converted to static entry
- Deleted after 30 seconds

👉 This avoids false positives while still protecting the network.

🆕 Enhancements from APIC 6.0(3)

Latest versions introduce more granular control:

New Capabilities

Create global rogue exception lists
Exclude specific MAC addresses from detection
Apply exclusions across:
- Bridge Domains
- L3Out networks

👉 This is very useful in:

Multi-tenant environments
Large-scale data centers
Automation-heavy environments

🚀 Real-World Use Case

Imagine a virtualized environment where VMs keep moving between hosts:

Without Rogue Detection:

Continuous MAC flapping
CPU spikes
Control-plane instability

With Rogue Detection:

Endpoint is quarantined
Stability is restored
Network continues to operate normally

💡 Best Practices

Always enable Rogue Endpoint Detection in production fabrics
Configure exception lists for:
- Load balancers
- VMware vMotion environments
Monitor faults regularly in APIC
Upgrade to ACI 3.2(6) or later for better behavior

Friday, 31 July 2026

The Problem: No VRF-Level BGP Timer Option in NDO

What We Asked Cisco TAC

Step-by-Step: Applying the BGP Timer Policy to a VRF via APIC

Why the Timers Won't Show Up Until You Reset the Session

Verifying the Change Actually Took Effect

A Word of Caution on Aggressive Timers

FAQ

Related Reading on Networklearner

Need help with Cisco ACI, Nexus, data center networking, or network automation?I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.Contact me for consulting, troubleshooting, design reviews, and project support.

Thursday, 23 July 2026

Friday, 26 June 2026

Cisco ACI vPC Explained: Architecture, Working, Benefits & Traffic Flow

Table of Contents

What is Cisco ACI vPC?

Why Do We Need vPC?

Traditional Nexus vPC vs Cisco ACI vPC

Cisco ACI vPC Architecture

Key Components of Cisco ACI vPC

1. Leaf Switches

2. Spine Switches

3. APIC Controller

4. LACP Port Channel

Understanding MCT Architecture

How Peer Communication Works

What is ZeroMQ (ZMQ)?

Understanding URIB

Benefits of Cisco ACI vPC

Coming Up in Part 2

📚 Related Cisco ACI Articles

1. Cisco ACI Explained – Concepts, Learning Prerequisites, Benefits & Interview Questions

2. Understanding VLAN Pool Roles in Cisco ACI

3. Understanding Domain Types in Cisco ACI

4. Key Concepts of Application Profile in Cisco ACI

5. Cisco ACI Static EPG Configuration – Step-by-Step Guide

6. Cisco ACI Port Channel Configuration (eth1/4 & eth1/5)

7. Configuring Port Profiles in Cisco ACI

8. L3Out Subnet Scope Options in Cisco ACI

9. What is a Contract Preferred Group in Cisco ACI?

Sunday, 26 April 2026

1. What Does “Unknown” Leaf State Mean in Cisco ACI?

2. Certificate Issues Between Leaf and APIC

3. LLDP Mismatch or LLDP Failure

4. Firmware or Software Incompatibility

5. Hardware Problems

6. Time Synchronization Issues

7. Incorrect Node ID or Serial Number Issues

8. Recommended Troubleshooting Sequence

9. Best Practices to Prevent Unknown Leaf State

10. Conclusion

Section 1: Basic Cisco ACI L3Out Interview Questions

1. What is L3Out in Cisco ACI?

2. Why do we need L3Out?

3. Is L3Out mandatory in ACI?

4. Where is L3Out configured?

5. Is L3Out Layer‑2 or Layer‑3?

Section 2: L3Out Components Interview Questions

6. What are the main components of L3Out?

7. What is a Logical Node Profile?

8. What is a Logical Interface Profile?

9. Can L3Out be deployed on multiple leafs?

10. What happens if an L3Out leaf fails?

Section 3: L3Out and Routing Protocol Interview Questions

11. Which routing protocols are supported with L3Out?

12. Which routing protocol is most commonly used?

13. Is OSPF supported in L3Out?

14. Can static routes be used in L3Out?

15. Can L3Out support ECMP?

Section 4: L3Out and VRF Association Questions

16. Is L3Out associated with a VRF?

17. Can one L3Out be shared across multiple VRFs?

18. Can multiple L3Outs exist in the same VRF?

19. Why would you create multiple L3Outs in one VRF?

20. What happens if VRF association is wrong?

Section 5: External EPG Interview Questions

21. What is an External EPG?

22. Why is an External EPG required?

23. How is traffic allowed between internal EPGs and External EPGs?

24. Is External EPG similar to internal EPG?

25. Can there be multiple External EPGs under one L3Out?

Need help with Cisco ACI, Nexus, data center networking, or network automation?
I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.
Contact me for consulting, troubleshooting, design reviews, and project support.