Showing posts with label Data Center Networking. Show all posts
Showing posts with label Data Center Networking. Show all posts

Wednesday, 5 August 2026

How AI Memory Can Revolutionize Cisco ACI Operations

Cisco ACI has transformed data center networking by introducing policy-based automation, centralized management, and application-centric networking. However, as enterprise fabrics grow, troubleshooting becomes increasingly challenging.

Imagine receiving multiple faults from different leaf switches after a maintenance window. Instead of manually checking APIC health, interface counters, contracts, endpoint learning, and event logs one by one, an AI assistant could analyze everything in seconds.

But how can an AI assistant remember previous troubleshooting steps, understand your fabric, and retrieve your organization's deployment standards?

The answer lies in AI Memory.

Just like an experienced Cisco ACI engineer relies on operational knowledge, design documents, and previous troubleshooting experience, AI agents use different types of memory to provide intelligent assistance.

In this article, we'll explore how AI memory works using real Cisco ACI scenarios.

Why Memory Matters in Cisco ACI

Suppose your production ACI fabric consists of:

  • Three APIC controllers
  • Two Spine switches
  • Twenty Leaf switches
  • Hundreds of EPGs
  • Multiple VRFs
  • External L3Out connections
  • VMware VMM integration

Now imagine that an application suddenly loses connectivity.

Without memory, an AI assistant would ask the same questions every time:

  • Which tenant is affected?
  • Which EPGs are involved?
  • What contracts exist?
  • Which leaf switch hosts the endpoint?
  • Was any policy recently modified?

An experienced engineer already remembers much of this context. Likewise, AI memory allows an intelligent assistant to retain relevant information, retrieve documentation, and personalize future troubleshooting.

The Three Types of AI Memory

Think of AI memory as the way a senior Cisco ACI architect organizes information.

AI MemoryCisco ACI Example
Working MemoryCurrent APIC fault investigation
Long-Term MemoryACI design documents and runbooks
Episodic MemoryPrevious incidents and administrator preferences

Together, these memory types help AI solve problems faster and more accurately.

Working Memory – The Current Troubleshooting Session

Working memory contains information that the AI agent is actively processing during the current investigation. It is temporary and limited to the ongoing session.

Cisco ACI Example

An engineer asks:

"Why are endpoints in EPG-Web unable to communicate with EPG-App?"

The AI assistant keeps track of:

  • Tenant name
  • VRF
  • Bridge Domain
  • EPG names
  • Applied contracts
  • Recent APIC fault messages
  • Results of previous checks

Instead of asking for the same information repeatedly, it builds on the conversation until the issue is resolved.

Long-Term Memory – Your Cisco ACI Knowledge Base

Long-term memory stores persistent information that does not disappear after the conversation ends. In AI systems, this often includes documentation and runbooks that can be retrieved when needed.

For Cisco ACI, this could include:

  • Fabric architecture diagrams
  • Tenant standards
  • Naming conventions
  • Interface policies
  • L3Out design guides
  • APIC backup procedures
  • Security policies
  • Change management documents

Cisco ACI Example

An engineer asks:

"What is our standard configuration for external routed networks?"

Instead of relying on generic knowledge, the AI searches your organization's approved ACI design guide and provides recommendations based on your own standards.

This improves consistency and reduces the risk of configuration drift.

What is RAG?

RAG (Retrieval-Augmented Generation) allows an AI assistant to retrieve relevant documentation before generating a response. Rather than guessing, it searches trusted sources such as your ACI runbooks, design guides, and operational procedures.

Cisco ACI Example

You ask:

"How do we normally configure BGP authentication on our L3Outs?"

The AI retrieves the organization's approved implementation guide and answers using that document rather than generic internet advice.

Episodic Memory – Learning from Previous Incidents

Episodic memory records previous interactions and user preferences so future assistance can be more personalized.

Cisco ACI Example

Suppose that last month you resolved a fault caused by a missing contract between two EPGs.

Months later, a similar fault occurs.

The AI recognizes the similarity and suggests checking contracts early in the troubleshooting process, saving valuable time.

It may also remember that you prefer CLI outputs alongside APIC GUI navigation, allowing responses to match your working style.

Bringing It All Together

Imagine a production outage affecting application connectivity.

An AI assistant could:

  1. Use Working Memory to keep track of the current troubleshooting session.
  2. Use Long-Term Memory to retrieve your organization's ACI standards and runbooks.
  3. Use Episodic Memory to recognize similar past incidents and apply successful troubleshooting patterns.

This combination provides faster diagnostics, more consistent recommendations, and reduced troubleshooting time.

Benefits for Cisco ACI Engineers

By combining AI memory with Cisco ACI, organizations can:

  • Accelerate root cause analysis.
  • Reduce repetitive troubleshooting.
  • Retrieve design documentation instantly.
  • Improve adherence to operational standards.
  • Preserve knowledge from experienced engineers.
  • Shorten onboarding time for new team members.
  • Enable more intelligent AI-driven network operations.

Key Takeaways

  • Working Memory manages the current troubleshooting context.
  • Long-Term Memory stores ACI documentation, standards, and runbooks.
  • Episodic Memory captures previous incidents and user preferences.
  • RAG connects AI to trusted enterprise documentation instead of relying only on model training.
  • Together, these capabilities can significantly improve Cisco ACI operations and troubleshooting efficiency.

Frequently Asked Questions

Can AI remember my Cisco ACI fabric permanently?
Only if the AI platform is designed to store and retrieve persistent knowledge such as documentation and previous interactions.

How does RAG help Cisco ACI administrators?
It allows AI to search approved ACI documentation and generate answers based on your organization's standards instead of generic information.

Can AI replace Cisco ACI engineers?
No. AI augments engineers by reducing repetitive tasks and surfacing relevant information, while design decisions and operational oversight remain with experienced professionals.


Related Articles to add to "How AI Memory Can Revolutionize Cisco ACI Operations":

Need help with Cisco ACI, Nexus, data center networking, or network automation?

I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.

Contact me for consulting, troubleshooting, design reviews, and project support.


Posted by at No comments:
Labels: , , , , , , , , ,

Friday, 31 July 2026

BGP Timers on a VRF in ACI Multi-Site — Why NDO Can't Do It and What Cisco TAC Confirmed

 If you've gone looking for a VRF-level BGP Timer Policy in Nexus Dashboard Orchestrator and come up empty, you're not missing a menu — it genuinely isn't there yet. Here's what we found while troubleshooting this on a live Multi-Site fabric with a mix of stretched and local VRFs, the workaround Cisco TAC confirmed as supported, and exactly how to verify the new timers actually took effect instead of just assuming they did.

  • The gap: NDO does not expose a BGP Timer Policy at the VRF level for ACI Multi-Site — only at the L3Out node level (via L3Out Node Routing Policy → BGP Node Settings).
  • The workaround: Create and apply the BGP Timer Policy directly on APIC, per site, against the target VRF (local or stretched). Cisco TAC confirmed this is the correct and supported method.
  • The catch: BGP timers are negotiated during the initial handshake. Changing the policy alone does nothing to an already-established session — you need a hard reset of the BGP session for the new keepalive/hold values to take effect.
  • Don't trust the config alone. "Configured" timers and "negotiated" (active) timers are two different things in show bgp output — always verify the negotiated value after the reset, not just the configuration.

The Problem: No VRF-Level BGP Timer Option in NDO

In a single-site APIC deployment, applying a custom BGP Timer Policy to a VRF is straightforward — it's a native object under Tenant → Networking → VRF → BGP Context Per Address Family. In Multi-Site environments managed through NDO, that granularity isn't there. NDO's L3Out configuration model lets you attach a BGP Node Routing Policy (with its own keepalive/hold timers) to a node group inside an L3Out template — but there's no equivalent construct for applying timers at the VRF level, independent of a specific L3Out's node groups.

For environments running mixed local and stretched VRFs across sites, that's a real gap: you may want consistent BGP timer behavior tied to the VRF itself, not scattered across every L3Out node group that happens to reference it.

What We Asked Cisco TAC

Rather than guess, we opened a case and asked directly. Here's the exchange, included for anyone hitting the same wall:

Our question:

We need to deploy BGP timers on a VRF within our ACI Multi-Site fabric. Our environment contains both stretched and local VRFs. We do not see an option in NDO to apply BGP timers at the VRF level. Therefore, we are planning to use the following approach: 1) Create a BGP Timer Policy on the APIC. 2) Apply the BGP Timer Policy to the target VRF (local or stretched) via APIC only. Could you please confirm whether the above procedure is the correct and supported method for configuring BGP timers in an ACI Multi-Site environment? Additionally, after applying the policy, we do not see the updated timer values reflected on the BGP neighbour sessions. Is a BGP session reset (hard reset) required for the new timer values to take effect, or are there any additional steps that need to be performed?

Cisco's response:

Configuration Method: Your proposed approach is correct. As NDO does not currently support configuring BGP Timer Policies at the VRF level, applying these policies directly via the APIC on each respective site is the standard and supported method. 

Session Reset: Yes, a BGP session reset (hard reset) is required for the new timer values to take effect. Because these timers are negotiated during the initial BGP handshake, the existing sessions must be torn down and re-established to renegotiate the new keepalive and hold-down values with the neighbor.

Two things worth underlining from that response: this is a per-site, APIC-side configuration — NDO is not the tool for VRF-level BGP timers today — and a hard reset is mandatory, not optional, for the values to actually apply.

Step-by-Step: Applying the BGP Timer Policy to a VRF via APIC

  1. Log into the APIC GUI for the site containing the target VRF (do this per site for stretched VRFs — there's no single-pane push for this from NDO).
  2. Navigate to Tenants → <tenant> → Networking → VRFs → <vrf-name>.
  3. Under the VRF, go to the BGP Context Per Address Family (create one for each address family in use — IPv4 and/or IPv6).
  4. Set the BGP Timers field to a custom BGP Timer Policy, or create a new one:
    • Go to Tenants → <tenant> → Policies → Protocol → BGP → BGP Timers, right-click and Create BGP Timers Policy.
    • Set Keepalive Interval (sec) and Hold Interval (sec) to your target values (keep the standard 1:3 ratio — e.g., 4/12 or 10/30 — most implementations expect hold time to be roughly 3x keepalive).
  5. Apply the policy to the VRF's BGP Context Per Address Family and submit.
  6. Repeat per site for any stretched VRF — this is not propagated automatically across the Multi-Site domain since it's an APIC-local object, not an NDO-managed one.
  7. Confirm the object landed correctly:
   GET https://<apic>/api/node/mo/uni/tn-<tenant>/ctx-<vrf>/bgpCtxAfP-ipv4-ucast.json

Check for the tnBgpTimersPolName attribute pointing to your new policy.

Why the Timers Won't Show Up Until You Reset the Session

This trips people up every time: BGP keepalive and hold timers are exchanged in the OPEN message during session establishment — they're not a live, continuously-enforced parameter that updates on the fly. Changing the policy updates the configuration, but an already-established session keeps running on whatever timers it originally negotiated until it's torn down and rebuilt.

That means after step 7 above, show bgp will show your configured timers, but the negotiated/active timers on any existing session will still be the old values — until you force a reset.

Verifying the Change Actually Took Effect

Don't just trust the APIC MO or the CLI's "configured" line — verify the negotiated value, which is what's actually governing the session.

1. Check the configured value on the border leaf:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

Look for: Configured hold time is 12, keepalive interval is 4 seconds

2. Check the negotiated (active) value in the same output: Look for: Hold time is X, keepalive interval is Y seconds If X/Y still show the old defaults, the session hasn't renegotiated — move to step 3.

3. Force the hard reset:

clear bgp <afi> unicast <peer-ip> vrf <tenant>:<vrf-name>

A soft clear will not renegotiate timers — timers only get exchanged at session establishment, so this has to be a hard clear.

4. Re-check the negotiated value after the reset:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

The negotiated value will be the lower of what each side offers — if your peer's own BGP timer configuration is set higher (or lower) than yours, that governs the final negotiated result, not your configured value alone.

5. Confirm real-world keepalive cadence:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last read"

Run this two or three times a few seconds apart — "Last read" should reset roughly every N seconds, matching your new keepalive interval.

6. Watch for flapping after the change, especially with aggressive timers like 4/12:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last reset"
show logging logfile | include BGP

A Word of Caution on Aggressive Timers

Tighter timers (like 4-second keepalive / 12-second hold) improve failure-detection speed, but they raise the risk of false-positive flaps on links with variable latency, high CPU load on the peer device, or congested control-plane paths. Before rolling this out broadly across a Multi-Site fabric, test it on a single non-critical VRF/L3Out first and monitor for a few days.

FAQ

Does NDO support VRF-level BGP timers at all? Not currently. The only BGP timer construct exposed in NDO is at the L3Out node group level (L3Out Node Routing Policy → BGP Node Settings). VRF-level BGP Timer Policies must be applied directly on APIC, per Cisco TAC's confirmation.

Will this survive an NDO template push or resync? Since the VRF-level BGP Timers Policy is applied directly on APIC and not modeled in the NDO template, it should not be overwritten by a standard template redeploy — but any workflow that deletes and recreates the VRF from NDO would remove it. Treat it as an out-of-band configuration and document it clearly for your team.

Do I need to reset the session on both sides of the peering, or just my side? Just your side's session needs a hard clear to force renegotiation — BGP will tear down and rebuild the full session, which inherently re-triggers the handshake and timer exchange on both ends.

What's a safe keepalive/hold ratio to use? Cisco and most vendors expect hold time to be roughly 3x keepalive (e.g., 4/12, 10/30, 20/60). Going below that ratio, or setting a very low hold time on a link with real latency or jitter, increases false-positive flap risk.

Is this the same procedure for a stretched VRF across multiple sites? Yes, but it must be repeated per site — this is an APIC-local object, and NDO doesn't propagate it across the Multi-Site domain automatically. For a stretched VRF, log into each site's APIC and apply the same BGP Timer Policy independently.

Related Reading on Networklearner

Author note: 

Need help with Cisco ACI, Nexus, data center networking, or network automation?
I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.
Contact me for consulting, troubleshooting, design reviews, and project support.

Posted by at No comments:
Labels: , , , , , , , , , , ,

Wednesday, 22 July 2026

Complete Firewalls Port Reference for Cisco ACI, Nexus Dashboard, NDO and NDI

 

If you've ever tried to onboard an APIC site into Nexus Dashboard Orchestrator (NDO) or Nexus Dashboard Insights (NDI) only to watch the connection sit at "Not Reachable," the cause is almost always the same: a firewall or ACL between Nexus Dashboard and the ACI fabric is blocking a port nobody told you about.

This guide consolidates every port you need — APIC to leaf/spine, APIC to Nexus Dashboard, NDO-specific ports, NDI-specific ports, and the internal ports a multi-node ND cluster needs between its own nodes — into one reference you can hand straight to your firewall team.

Quick jump:

Why this trips people up

Nexus Dashboard isn't one service — it's a platform that runs NDO, NDI, and NDFC as apps on top of a shared cluster. Each app has its own port requirements on top of the base cluster ports, and each app can reach the fabric through a different interface (management network vs. data network, in-band vs. out-of-band APIC). Miss that distinction and you'll open the right port on the wrong interface — the connection still fails, and it looks identical to a wrong-port problem.

Management vs. data network — pick correctly {#management-vs-data-network}

Nexus Dashboard interfaceWhat it's forReaches APIC via
Management (mgmt) networkUI access, NTP, DNS, firmware upgrades, Intersight, DC proxyCan reach APIC OOB or in-band, if routed
Data networkNode-to-node cluster traffic, and all app traffic — NDO, NDI, NDFCAPIC in-band or OOB for NDO; in-band only for NDI

Rule of thumb: if you're only running NDI, or NDI alongside other services, the ND data interface must have IP reachability to the in-band management network of the APIC and every switch in the fabric — NDI pulls switch-level telemetry that OOB simply doesn't carry. NDO is more flexible and can use either OOB or in-band APIC reachability.

Port table: ND cluster ↔ outside world {#nd-cluster-ports}

These are baseline ports every Nexus Dashboard cluster needs, regardless of which app (NDO/NDI/NDFC) you run on top.



Also allow outbound to your NTP, DNS, and (if used) TACACS+/LDAP/RADIUS servers, plus internet/proxy access if the cluster needs firmware downloads or Intersight connectivity.

Port table: NDO ↔ APIC {#ndo-apic-ports}



NDO's actual data-plane traffic to sites is just the APIC REST API over TCP 443. If you're only building day-0/day-1 policy through NDO (no NDI), this is the shortest port list of the three.

Port table: NDI ↔ APIC / fabric {#ndi-apic-ports}


NDI is the most sensitive to the in-band requirement — before onboarding a fabric, confirm the in-band management EPG, bridge domain, and subnet are already built and that every leaf, spine, and APIC has a reachable in-band address. This is the step people skip and then spend hours troubleshooting "site unreachable" errors that are actually EPG/BD issues, not firewall issues.

Port table: APIC ↔ Leaf/Spine (fabric internal) {#apic-fabric-ports}

This traffic stays inside the ACI infra VRF and normally isn't firewalled, but it's worth documenting for anyone running APIC connectivity through an external device (e.g., a device sitting between APIC and a remote-leaf pair):

If you're extending remote leaf switches over a routed IPN, also confirm OSPF, DHCP relay, and multicast (PIM Bidir) are permitted, with a minimum MTU of 9150 bytes end to end.

Troubleshooting checklist {#troubleshooting}

Work through these in order before opening a TAC case:

  1. Ping test first. Confirm basic IP reachability from the ND data interface to the APIC in-band (for NDI) or in-band/OOB (for NDO) address. No ping = no point checking ports yet.
  2. Confirm which interface ND is actually using. Nexus Dashboard > Infrastructure > Cluster Configuration shows whether the data or management network is used for fabric connectivity.
  3. Verify APIC in-band is actually built (mgmt tenant, in-band BD, in-band EPG, node management addresses) — this is the single most common NDI onboarding blocker, not the firewall itself.
  4. Test port 443 directly from an ND node to the APIC IP with curl -vk https://<apic-ip> or openssl s_client -connect <apic-ip>:443.
  5. Check for SSL inspection/proxy devices sitting in the path — APIC's certificate handling doesn't play well with transparent SSL interception.
  6. Re-check site status after any firewall change — NDO/NDI cache connectivity state and may need a manual "Refresh" or re-registration, not just a rule change.

FAQ {#faq}

Does NDO need port 80, or just 443? In current releases, TCP 443 alone is sufficient for APIC REST API communication. Port 80 was referenced in older MSO documentation and is generally not required if 443 is reachable.

Can NDI use the APIC out-of-band address? No. NDI depends on in-band reachability to both APIC and the switches for telemetry collection; OOB-only connectivity will not work for NDI.

Do NDO and NDI need different firewall rules if both run on the same ND cluster? Yes — treat them as separate rule sets. NDO needs 443 to APIC. NDI additionally needs 443 (and sometimes 22) to every switch in the fabric over the in-band network.

What's the minimum MTU I need between Nexus Dashboard and the fabric? 1500 bytes minimum on the ND data interface; higher MTU is supported if your infrastructure already runs jumbo frames.

My rules look correct but the site still shows "Not Reachable." What next? Check for SSL-inspecting firewalls/proxies in the path, confirm the APIC certificate hasn't expired, and verify NTP is in sync across APIC and ND — clock skew alone can break TLS session establishment.

Related reading on Networklearner

Posted by at No comments:
Labels: , , , , , , , , ,

Tuesday, 12 May 2026

How to Remove VLANs from OTV on Cisco Nexus 7000 (N7K) Step by Step Guide

Introduction
Overlay Transport Virtualization (OTV) is used to extend Layer 2 VLANs across different data center sites. During migration, cleanup, or service decommissioning, you may need to remove VLANs from OTV. This must be done carefully to avoid impacting live traffic.

This guide explains how to remove all VLANs or selected VLANs from OTV using correct syntax and best practices.

Remove All VLANs from OTV
To remove all VLANs configured under OTV, use the following command:

N7K(config-if-overlay)# no otv extend-vlan

Example:

N7K(config-if-overlay)# no otv extend-vlan 2-100

This command removes all VLANs from being extended across the OTV overlay.

Remove Selected VLANs from OTV
Important point: Cisco Nexus OTV does not allow removing a VLAN directly from a range. You must redefine the VLAN list.

Scenario 1: Remove VLANs from a Range

Current configuration:

otv extend-vlan 2-34

Requirement: remove VLAN 3 and VLAN 4

Correct approach:

N7K(config-if-overlay)# otv extend-vlan 2,5-34

After this change, VLAN 3 and VLAN 4 will no longer be extended across OTV.

Or

otv extend-vlan remove 10

Verification Commands

After making changes, verify the configuration:

show otv vlan
show otv route

To check if a VLAN is actively used:

show mac address-table vlan

Impact of VLAN Removal

Before removing a VLAN, consider the following:

  • Layer 2 extension for that VLAN will stop
  • Traffic between data centers for that VLAN will be impacted
  • Applications using that VLAN may go down

Best Practices

  • Always verify VLAN usage before removal
  • Coordinate with application and network teams
  • Perform changes during maintenance window
  • Keep rollback configuration ready

Rollback Command

If required, restore the original VLAN list:

otv extend-vlan

Conclusion

Removing VLANs from OTV requires proper planning and correct command usage. While removing all VLANs is simple, removing specific VLANs requires redefining the VLAN list. Following best practices ensures smooth execution without impacting production services.

If you want, I can create more SEO and AdSense optimized blogs on Cisco ACI, Nexus, BGP, or real troubleshooting scenarios.

Posted by at No comments:
Labels: , , , , ,

Wednesday, 6 May 2026

Cisco DNA Center Enterprise vs Cluster Interface: Architecture, Design, and Best Practices Explained

 

DNAC Interface Overview

Cisco DNA Center primarily uses two important interfaces:

  1. Enterprise Interface (ENT) – External communication
  2. Cluster Interface – Internal node communication

Both serve completely different purposes but are equally important.

Enterprise Interface (ENT)

What is Enterprise Interface?

The Enterprise Interface is the main external-facing interface of DNAC.

Enterprise  interface in DNAC is always deployed as Layer 3.

It is used by:

  • Network engineers (GUI access)
  • APIs and integrations
  • Network devices

Key Functions

The ENT interface is responsible for:

  • GUI access (HTTPS)
  • REST API communication
  • Device management (Switches, Routers, WLCs)
  • Integration with external services:
    • DNS
    • NTP
    • TACACS / RADIUS
    • Active Directory
  • Third-party integrations (ServiceNow, SIEM)

Key Characteristics

  • Routable IP address
  • Accessible from enterprise network
  • Carries management and telemetry traffic
  • User-facing interface
  • Mission-critical for operations

Design Best Practices

  • Ensure high availability
  • Apply proper firewall/security policies
  • Avoid congestion (affects automation & telemetry)
  • Place in secure DC/management VLAN

Cluster Interface

What is Cluster Interface?

The Cluster Interface is used for internal communication between DNAC nodes.

Cluster interface in DNAC is typically deployed as Layer 2 (same subnet/VLAN) — NOT routed (L3)

👉 This is only required in multi-node cluster deployments (e.g., 3-node cluster).

Key Functions

The cluster interface handles:

  • Node-to-node communication
  • Database replication
  • Application synchronization
  • Cluster health monitoring
  • Workload distribution

Key Characteristics

  • Not accessible by users
  • Used only internally
  • Typically non-routable or restricted
  • Requires low latency
  • Sensitive to packet loss

Design Best Practices

  • Use a dedicated network
  • Ensure low latency (same DC preferred)
  • Avoid WAN usage
  • Monitor bandwidth utilization
  • Keep it isolated for stability

Enterprise vs Cluster Interface (Comparison)

Enterprise Interface

  • External communication
  • GUI and API access
  • Device connectivity
  • Routable and user-facing
  • Required in all deployments

Cluster Interface

  • Internal communication
  • Node synchronization
  • Database replication
  • Private and restricted
  • Required only in multi-node clusters

Real-World Deployment Example

In a 3-node DNAC cluster:

  • Enterprise Network: 10.10.10.0/24
  • Cluster Network: 192.168.100.0/24

Traffic Flow

  • Users access GUI → Enterprise Interface
  • Devices communicate → Enterprise Interface
  • Node synchronization → Cluster Interface

Common Mistakes to Avoid

  • Using same network for ENT and Cluster
  • Deploying cluster communication over WAN
  • Ignoring latency requirements
  • Not securing ENT interface
  • Underestimating cluster bandwidth

Troubleshooting Guide

Check Enterprise Interface if:

  • GUI is not accessible
  • Devices are unreachable
  • API calls are failing

Check Cluster Interface if:

  • Cluster health is degraded
  • Nodes are out of sync
  • Database or service issues occur

Why This Matters

Incorrect interface design can cause:

  • Device onboarding failures
  • Automation issues
  • Data inconsistency
  • Cluster instability
  • Poor user experience

Final Thoughts

  • Enterprise Interface = External communication gateway
  • Cluster Interface = Internal DNAC backbone

Both interfaces play a crucial role in ensuring:

  • Stability
  • Scalability
  • High performance

Designing them correctly ensures a reliable Cisco DNA Center deployment.

Conclusion

Understanding the difference between ENT and Cluster interfaces is essential for every network engineer working with Cisco DNAC.

A well-designed interface architecture results in:

  • Better performance
  • Easier troubleshooting
  • Improved user experience
Posted by at No comments:
Labels: , , , , ,

Tuesday, 28 April 2026

Cisco ACI Leaf Node ID Swap on vPC Pairs: Risks, Precautions, and Step-by-Step Process

Cisco ACI Leaf Node ID Swap Steps When Leaves Are Part of vPC

Step 0 – Preconditions
Confirm maintenance window is approved. Ensure alternate connectivity or downtime is acceptable. Make sure you have console or OOB access to both leaf switches.

Step 1 – Drain Traffic and Clear Endpoints
Shut down or migrate all server-facing interfaces connected to the vPC pair.
From APIC, navigate to Fabric → Inventory → Pod → Node → Leaf → Endpoints.
Verify endpoint count is zero on both leaves.

Step 2 – Remove vPC and Port-Channel Configuration
Delete vPC protection group policies.
Delete all vPC port-channels.
Remove interface policy associations.
Remove all static EPG bindings that reference the vPC or either leaf.
At this stage, the leaves must have no access policy dependencies.

Step 3 – Remove L3Out (If Leaves Are Border Leaves)
If the vPC pair is used for L3Out, remove both leaves from the L3Out logical node profile and logical interface profile.
Confirm external routing is stable via remaining border leaves.

Step 4 – Decommission First Leaf (Leaf A)
In APIC, go to Fabric → Inventory → Fabric Membership.
Select Leaf A and perform Decommission.
Wait until the status shows Decommissioned.
Do not power off yet.

Step 5 – Clean Leaf A
Connect to Leaf A using console or OOB.
Run acidiag touch clean and then reload the switch.
This removes old node ID, certificates, and fabric identity.

Step 6 – Decommission Second Leaf (Leaf B)
In APIC, again go to Fabric → Inventory → Fabric Membership.
Select Leaf B and perform Decommission.
Wait until the status shows Decommissioned.

Step 7 – Clean Leaf B
Connect to Leaf B using console or OOB.
Run acidiag touch clean and reload the switch.
Both leaves are now clean and discovery-ready.

Step 8 – Re-add Leaf A with New Node ID
Power on Leaf A only.
Ensure fabric uplinks to spines are connected.
From APIC Fabric Membership, approve the switch and assign the new desired node ID.
Wait until Leaf A is fully discovered and stable.

Step 9 – Re-add Leaf B with New Node ID
Power on Leaf B.
From APIC Fabric Membership, approve it and assign the other node ID.
Wait until Leaf B is fully discovered and stable.

Step 10 – Rebuild vPC Configuration
After both leaves are healthy, recreate the vPC protection group.
Recreate vPC port-channels and interface policies.
Reapply static EPG bindings to the vPC.
Do not rebuild vPC if only one leaf is active.

Step 11 – Validation
Verify fabric health is green.
Ensure no vPC, access, or infra faults exist.
Confirm port-channels are up on both leaves.

Step 12 – Restore Traffic
Enable server-facing interfaces.
Bring servers or upstream devices back online.
Verify endpoint learning and confirm no MAC flapping or faults.

Final Rule

Never attempt a live node ID swap. Always decommission, clean, and re-add both vPC peer leaves in a controlled sequence.

Precautions

Swapping node IDs between Cisco ACI leaf switches is a sensitive operation, especially when the leaves are configured as a vPC pair. Unlike traditional networks, Cisco ACI tightly binds policies, forwarding state, and infrastructure objects to node IDs, making a node ID swap a planned maintenance activity, not a live change. When vPC is involved, the risk multiplies because both leaves act as a single logical endpoint for servers and network devices.

This article explains the critical precautions you must follow when performing a Cisco ACI leaf node ID swap in a vPC environment, based on real production experience and Cisco‑accepted operational practices.

Why Node ID Swap Is Risky in vPC‑Based ACI Fabrics

In Cisco ACI, a leaf’s node ID is not just an identifier; it is embedded into multiple internal constructs such as vPC identifiers, static EPG bindings, endpoint tables, and forwarding databases. In a vPC pair, both leaves jointly provide forwarding for a single logical port‑channel. Swapping node IDs without proper preparation can cause MAC flapping, endpoint blackholing, broken port‑channels, and fabric faults.

There is no supported in‑place node ID change in Cisco ACI. The only supported method to swap node IDs is to decommission, clean, and re‑add the leaf switches with the desired node IDs.

Precaution 1: Treat the vPC Pair as a Single Failure Domain

The most important rule is to treat both vPC peers as a single unit, even though they are two physical switches. Never attempt a node ID swap on only one vPC peer while the other peer is actively forwarding traffic. ACI vPC forwarding relies on consistent node information across both leaves. Any mismatch can result in unpredictable traffic loss.

Before starting, ensure:

  • All connected servers or upstream devices are drained or shut down.
  • No single‑homed devices depend on the vPC pair.
  • Maintenance is scheduled during a proper change window.

Precaution 2: Ensure Zero Active Endpoints on Both Leaves

A node ID swap must never be performed while endpoints are active. In ACI, endpoints can be learned dynamically through traffic, and their state is tied to the leaf node ID. If endpoints remain on either vPC peer, swapping node IDs will cause immediate disruption.

From APIC, verify that both leaves show zero endpoints before proceeding. If endpoints are present, migrate workloads, shut down interfaces, or disconnect cables until endpoint learning is cleared.

Precaution 3: Remove vPC and Port‑Channel Policies Before Decommissioning

ACI does not automatically clean up vPC policies during decommissioning. All vPC‑related constructs must be removed manually. This includes:

  • vPC protection group
  • Port‑channel policies
  • Interface policy associations
  • Static EPG bindings referencing the vPC

Leaving these objects in place can block decommissioning or result in orphaned configuration that causes faults after the swap. A clean policy removal ensures that the fabric does not retain references to the old node IDs.

Precaution 4: If the vPC Pair Is Also a Border Leaf, Remove L3Out First

When a vPC pair is serving as a border leaf for L3Out, the risk is even higher. External routing protocols such as BGP or OSPF depend on stable leaf identities. Before any node ID swap:

  • Remove the leaves from all L3Out logical node profiles.
  • Ensure routing is fully operational on alternate border leaves.
  • Validate external reachability before continuing.

Failure to do this can result in complete north‑south traffic outages.

Precaution 5: Always Clean Both Leaves Using acidiag

After decommissioning each leaf, it is mandatory to run:

acidiag touch clean
reload

on both vPC peers. Cleaning only one switch is a common and dangerous mistake. If one leaf still retains fabric identity or certificates, the fabric may encounter node ID conflicts, discovery failures, or inconsistent vPC behavior when the switches are re‑added.

Cleaning ensures that the switch boots in a discovery‑ready state with no residual ACI identity.

Precaution 6: Re‑Add Leaves Sequentially, Not in Parallel

When re‑adding switches with swapped node IDs, never power up or approve both leaves at the same time. Always follow a controlled order:

  1. Bring up the first leaf and assign its new node ID.
  2. Wait for full fabric stability and health.
  3. Bring up the second leaf and assign its new node ID.

This approach avoids node ID collisions, partial vPC instantiation, and confusing APIC fault scenarios.

Precaution 7: Rebuild vPC Only After Both Leaves Are Fully Healthy

Do not recreate vPC configurations until both leaves are fully discovered, healthy, and visible in the fabric. Building vPC with only one peer active leads to port‑channel inconsistencies and deployment failures.

Once both leaves are stable:

  • Recreate vPC protection groups.
  • Recreate port‑channels.
  • Reapply static EPG bindings.
  • Validate that both leaves appear in all bindings.

Only after this should server ports or network devices be reconnected.

Precaution 8: Validate vPC Health Before Allowing Traffic

Before reintroducing traffic, perform strict validation:

  • No vPC‑related faults in APIC.
  • Port‑channels show operational status.
  • No access, fabric, or infra faults.
  • Leaf interfaces are up and error‑free.

Once validation is complete, gradually restore server or upstream connectivity and observe endpoint learning behavior.

Common Mistakes to Avoid

The most common mistakes during node ID swap in vPC environments include attempting a live swap, forgetting to remove vPC policies, cleaning only one leaf, or restoring traffic before full validation. Each of these can result in extended outages and complex recovery procedures.

Final Takeaway

A Cisco ACI leaf node ID swap in a vPC environment is a full teardown and rebuild operation, not a minor change. Success depends on treating both leaves as a single unit, removing all dependencies, cleaning both switches, and performing a controlled re‑addition process. When executed correctly, the swap is safe and fully supported, but shortcuts almost always lead to problems.

One‑Line Summary

In Cisco ACI, swapping node IDs on vPC‑connected leaf switches requires full vPC teardown, clean decommissioning of both leaves, and a controlled rebuild to avoid traffic loss and fabric instability.

Posted by at No comments:
Labels: , , , , , , , , ,

Sunday, 26 April 2026

Cisco ACI “Unknown” Leaf State Explained: Certificates, LLDP, Software, and Hardware Issues

  In a Cisco ACI fabric, one of the most frustrating issues during initial fabric bring‑up, expansion, or node replacement is seeing a leaf switch stuck in an “Unknown” state. When a leaf is in an unknown state, it means the APIC cannot fully discover, authenticate, or manage the node, preventing it from joining the fabric and participating in traffic forwarding.

This issue can occur during initial fabric deployment, adding a new leaf to an existing fabric, replacing failed hardware, performing software upgrades, or moving switches between fabrics.

Understanding why a leaf enters the “Unknown” state is critical for fast recovery. In most cases, the root cause is not a single configuration mistake but a failure in communication, authentication, compatibility, or initialization.

This article explains the most common causes of the “Unknown” leaf state in Cisco ACI, why they happen, and how to systematically troubleshoot them in real‑world environments.

1. What Does “Unknown” Leaf State Mean in Cisco ACI?

When a leaf is shown as “Unknown” in the APIC GUI, it indicates that the APIC can see the node attempting discovery, but the node cannot complete secure authentication or critical control‑plane messaging has failed.

At this stage, the leaf is not operational, not programmable, and cannot forward production traffic.

2. Certificate Issues Between Leaf and APIC

Cisco ACI uses mutual certificate‑based authentication between the APIC controllers and fabric nodes. Every leaf switch must present a valid certificate chain that is signed and trusted by the APIC.

If the certificate exchange fails, the leaf cannot authenticate correctly, and APIC marks it as Unknown.

Common certificate‑related problems include an invalid or corrupted certificate on the leaf, the leaf previously belonging to another ACI fabric, expired or mismatched certificates due to time drift, or incomplete cleanup after node replacement.

These issues are often seen when hardware is reused without full re‑initialization.

The most reliable resolution is to completely wipe and reinitialize the leaf switch, ensure it boots in ACI mode, and allow APIC to generate and install a fresh certificate.

3. LLDP Mismatch or LLDP Failure

Cisco ACI relies heavily on LLDP for fabric discovery and adjacency validation. LLDP is mandatory in ACI for identifying correct topological relationships between leaf and spine switches.

If LLDP is not exchanged correctly, discovery fails and the leaf remains in an Unknown state.

Typical LLDP problems include LLDP being disabled on connected devices, LLDP filtered due to security policies, incorrect cabling such as connecting a leaf to something other than a spine, or the switch running in NX‑OS mode instead of ACI mode.

Symptoms include missing neighbor information, partial discovery, or interfaces appearing operationally down.

To resolve LLDP issues, ensure LLDP is enabled end‑to‑end, verify correct cabling from leaf to spine only, confirm the switch is running in ACI mode, and check optics and interfaces on both ends.

4. Firmware or Software Incompatibility

ACI fabric components are designed to work within a compatible software matrix. Significant software mismatches between the APIC, leaf, and spine can prevent successful node onboarding.

This often occurs when a leaf is running an unsupported ACI version, the APIC has been upgraded but the leaf image was not updated, or an incorrect software image is installed on the switch.

Typical symptoms include the leaf being detected but never transitioning from Unknown to Active, along with compatibility or image‑related faults.

Resolution requires verifying Cisco’s supported version matrix and ensuring that the leaf software version is compatible with both the APIC and spine versions.

5. Hardware Problems

Physical layer issues are a common but frequently overlooked cause of Unknown leaf state. Even a simple faulty optic can completely prevent discovery.

Common hardware causes include defective or unsupported transceivers, damaged fiber or copper cables, faulty ports on the leaf or spine, or mismatched speed or media types.

Indicators include interfaces staying down, intermittent connectivity, missing LLDP information, or hardware‑related faults in APIC.

Troubleshooting involves replacing suspect cables and optics, using only Cisco‑supported transceivers, testing alternate ports, and validating interface status on both leaf and spine.

6. Time Synchronization Issues

Certificate validation in ACI is time‑sensitive. If the system time on the leaf is significantly out of sync with the APIC, certificate authentication can fail even if the configuration and connectivity are correct.

This is common in environments where NTP is misconfigured, unavailable, or the device has been powered off for an extended period.

Symptoms include authentication failures and persistent Unknown leaf state with no obvious physical or configuration issues.

Resolution involves verifying NTP configuration on APIC, ensuring the leaf can synchronize time, and reinitiating discovery after time correction.

7. Incorrect Node ID or Serial Number Issues

ACI uniquely identifies nodes using a combination of node ID, serial number, and certificates. If these identifiers do not match what APIC expects, the leaf will fail authentication.

This commonly occurs when a switch was previously part of another fabric, reused after RMA without proper cleanup, or when a node ID conflict exists.

Symptoms include the leaf appearing with unexpected identity information or being rejected during registration.

The safest resolution is to fully wipe the leaf configuration, reboot the device, and allow APIC to assign a fresh node identity.

8. Recommended Troubleshooting Sequence

When a leaf is stuck in Unknown state, follow this sequence:

First, verify physical connectivity and optics.
Second, confirm LLDP adjacency and cabling.
Third, check software compatibility.
Fourth, validate certificates and authentication.
Fifth, ensure correct time synchronization.
Finally, reinitialize the leaf if needed.

Following this order avoids unnecessary configuration changes and reduces downtime.

9. Best Practices to Prevent Unknown Leaf State

Always wipe reused hardware before deployment.
Keep APIC, spine, and leaf software versions compatible.
Use supported Cisco optics and cables.
Ensure stable NTP configuration.
Verify LLDP connectivity during installation.
Document node IDs and serial numbers carefully.

Most Unknown leaf issues are preventable with proper procedures.

10. Conclusion

An Unknown leaf state in Cisco ACI is always a symptom of a failed discovery, authentication, compatibility, or initialization process. Certificate issues, LLDP failures, firmware incompatibility, hardware problems, time synchronization issues, and incorrect node identity are the most common causes.

By understanding these root causes and following a structured troubleshooting approach, engineers can resolve Unknown leaf issues quickly and avoid prolonged deployment delays.

A clean initialization and methodical verification remain the most effective solution in Cisco ACI environments.

Posted by at No comments:
Labels: , , , , , , , , ,

Cisco ACI L3Out Interview Questions Explained – Design, Implement, and Troubleshooting

  

Section 1: Basic Cisco ACI L3Out Interview Questions

1. What is L3Out in Cisco ACI?

L3Out (Layer‑3 Outside) is the ACI construct that provides external Layer‑3 connectivity between the ACI fabric and networks outside the fabric.

2. Why do we need L3Out?

L3Out is used to:

  • Connect ACI to external routers
  • Integrate firewalls
  • Provide north‑south traffic
  • Advertise routes between ACI and external networks

3. Is L3Out mandatory in ACI?

No. L3Out is required only if the ACI fabric needs external Layer‑3 communication.

4. Where is L3Out configured?

L3Out is configured under a Tenant, associated with a VRF, and deployed on leaf switches.

5. Is L3Out Layer‑2 or Layer‑3?

L3Out is strictly a Layer‑3 construct.

Section 2: L3Out Components Interview Questions

6. What are the main components of L3Out?

  • L3Out object
  • Logical Node Profile
  • Logical Interface Profile
  • External EPG
  • Contracts

7. What is a Logical Node Profile?

It defines which leaf nodes participate in the L3Out.

8. What is a Logical Interface Profile?

It defines:

  • Interface type (routed, SVI)
  • IP addressing
  • Encapsulation (VLAN)
  • Connectivity to external device

9. Can L3Out be deployed on multiple leafs?

Yes. L3Out is commonly deployed on multiple leaf switches for redundancy.

10. What happens if an L3Out leaf fails?

Traffic fails over to other L3Out‑enabled leafs, assuming proper design (ECMP / routing).

Section 3: L3Out and Routing Protocol Interview Questions

11. Which routing protocols are supported with L3Out?

  • Static routing
  • OSPF
  • BGP

12. Which routing protocol is most commonly used?

BGP, due to scalability and flexibility.

13. Is OSPF supported in L3Out?

Yes, but less commonly used in large deployments.

14. Can static routes be used in L3Out?

Yes, for simple or small environments.

15. Can L3Out support ECMP?

Yes. ACI supports ECMP for L3Out when routing protocols allow it.

Section 4: L3Out and VRF Association Questions

16. Is L3Out associated with a VRF?

Yes. Every L3Out must be associated with exactly one VRF.

17. Can one L3Out be shared across multiple VRFs?

No. One L3Out belongs to only one VRF.

18. Can multiple L3Outs exist in the same VRF?

Yes. A VRF can have multiple L3Outs.

19. Why would you create multiple L3Outs in one VRF?

  • Multiple external devices
  • Separate routing domains
  • Different security or routing policies

20. What happens if VRF association is wrong?

External routing will fail and traffic will be dropped.

Section 5: External EPG Interview Questions

21. What is an External EPG?

An External EPG represents external networks outside the ACI fabric.

22. Why is an External EPG required?

Because ACI is deny‑by‑default, and external networks must also follow ACI security policy.

23. How is traffic allowed between internal EPGs and External EPGs?

Using contracts.

24. Is External EPG similar to internal EPG?

Conceptually yes, but it represents external endpoints.

25. Can there be multiple External EPGs under one L3Out?

Yes.

Section 6: L3Out and Contracts (Very Important)

26. Is traffic allowed by default between ACI and external networks?

No. Traffic is denied by default.

27. How do you allow internal traffic to external networks?

Apply contracts between internal EPG and External EPG.

28. Can External EPG be provider or consumer?

It can be either or both, depending on traffic flow.

29. What happens if no contract is applied?

Traffic will be dropped, even though routing is correct.

30. Why do many L3Out issues occur?

Because routing works, but contracts are missing or incorrect.

Section 7: L3Out Design Interview Questions

31. Routed Interface vs SVI – what is preferred?

Routed interfaces are preferred for simplicity and scale.

32. When would you use SVI‑based L3Out?

When connecting to:

  • Traditional VLAN‑based networks
  • Legacy firewalls

33. Can L3Out connect to firewalls?

Yes, very commonly.

34. Can one firewall connect to multiple L3Outs?

Yes, depending on design.

35. Should L3Out be deployed on border leafs?

Yes. Border leafs are best practice.

Section 8: Advanced L3Out Interview Questions

36. How is route leaking handled in ACI?

Using Shared Services VRF and contracts.

37. Can L3Out be used with Shared Services VRF?

Yes, very commonly.

38. Can L3Out be stretched across sites?

  • Multi‑Pod: Yes
  • Multi‑Site: Via individual site L3Outs

39. How does L3Out behave in Multi‑Pod?

L3Out is shared across pods.

40. How does L3Out behave in Multi‑Site?

Each site has its own L3Out, orchestrated by NDO.

Section 9: L3Out and External Connectivity Troubleshooting Questions

41. Routing is correct but traffic fails – why?

Most likely contract or filter issue.

42. Endpoint can ping gateway but not internet – why?

External EPG contract missing or incorrect.

43. How to verify routes learned from L3Out?

  • APIC routes view
  • Leaf show commands
  • moquery

44. How do you verify contract programming?

Use:

show zoning-rule

45. How do you verify L3Out operational status?

  • APIC Health score
  • Faults
  • Leaf CLI

Section 10: MoQuery Commands for L3Out Verification

46. Verify L3Out configuration

moquery -c l3extOut

47. Verify External EPGs

moquery -c l3extInstP

48. Verify L3Out subnets

moquery -c l3extSubnet

49. Verify VRF association

moquery -c fvCtx

50. Check faults related to L3Out

moquery -c faultInst

Section 11: Common L3Out Mistakes (Interview Favorite)

51. Forgetting contracts

Most common mistake.

52. Wrong VRF association

Causes route blackholing.

53. Deploying L3Out on wrong leaf

Traffic won’t exit properly.

54. Using SVI instead of routed interface unnecessarily

Adds complexity.

55. Not planning for redundancy

Leads to single‑point failures.

Section 12: Scenario‑Based L3Out Interview Questions

56. When should you create multiple External EPGs?

When different external networks need different security policies.

57. Can multiple L3Outs advertise the same prefix?

Yes, but routing behavior must be carefully designed.

58. Can L3Out connect to non‑Cisco devices?

Yes. ACI is vendor‑agnostic at Layer‑3.

59. Can L3Out be used for Internet access?

Yes, with proper NAT/firewall integration.

60. What is the biggest design challenge in L3Out?

Balancing security, simplicity, and scalability.

Conclusion

Cisco ACI L3Out is the gateway between the ACI fabric and the external world. Interviews around L3Out focus on design understanding, security enforcement, VRF association, and troubleshooting approach, not just configuration steps.

If you understand:

  • How routing works
  • Why contracts are mandatory
  • Where L3Out should be placed
  • How to verify and troubleshoot

you will handle most Cisco ACI L3Out interview questions confidently.

✅ Interview Tip

When answering L3Out questions, always explain:

  1. Routing
  2. Security (contracts)
  3. Placement (leafs)
  4. Verification
Posted by at No comments:
Labels: , , , , , , , , , ,
Subscribe to: Posts (Atom)