BGP Timers on a VRF in ACI Multi-Site — Why NDO Can't Do It and What Cisco TAC Confirmed

 If you've gone looking for a VRF-level BGP Timer Policy in Nexus Dashboard Orchestrator and come up empty, you're not missing a menu — it genuinely isn't there yet. Here's what we found while troubleshooting this on a live Multi-Site fabric with a mix of stretched and local VRFs, the workaround Cisco TAC confirmed as supported, and exactly how to verify the new timers actually took effect instead of just assuming they did.

• The gap: NDO does not expose a BGP Timer Policy at the VRF level for ACI Multi-Site — only at the L3Out node level (via L3Out Node Routing Policy → BGP Node Settings).
• The workaround: Create and apply the BGP Timer Policy directly on APIC, per site, against the target VRF (local or stretched). Cisco TAC confirmed this is the correct and supported method.
• The catch: BGP timers are negotiated during the initial handshake. Changing the policy alone does nothing to an already-established session — you need a hard reset of the BGP session for the new keepalive/hold values to take effect.
• Don't trust the config alone. "Configured" timers and "negotiated" (active) timers are two different things in show bgp output — always verify the negotiated value after the reset, not just the configuration.

The Problem: No VRF-Level BGP Timer Option in NDO

In a single-site APIC deployment, applying a custom BGP Timer Policy to a VRF is straightforward — it's a native object under Tenant → Networking → VRF → BGP Context Per Address Family. In Multi-Site environments managed through NDO, that granularity isn't there. NDO's L3Out configuration model lets you attach a BGP Node Routing Policy (with its own keepalive/hold timers) to a node group inside an L3Out template — but there's no equivalent construct for applying timers at the VRF level, independent of a specific L3Out's node groups.

For environments running mixed local and stretched VRFs across sites, that's a real gap: you may want consistent BGP timer behavior tied to the VRF itself, not scattered across every L3Out node group that happens to reference it.

What We Asked Cisco TAC

Rather than guess, we opened a case and asked directly. Here's the exchange, included for anyone hitting the same wall:

Our question:

We need to deploy BGP timers on a VRF within our ACI Multi-Site fabric. Our environment contains both stretched and local VRFs. We do not see an option in NDO to apply BGP timers at the VRF level. Therefore, we are planning to use the following approach: 1) Create a BGP Timer Policy on the APIC. 2) Apply the BGP Timer Policy to the target VRF (local or stretched) via APIC only. Could you please confirm whether the above procedure is the correct and supported method for configuring BGP timers in an ACI Multi-Site environment? Additionally, after applying the policy, we do not see the updated timer values reflected on the BGP neighbour sessions. Is a BGP session reset (hard reset) required for the new timer values to take effect, or are there any additional steps that need to be performed?

Cisco's response:

Configuration Method: Your proposed approach is correct. As NDO does not currently support configuring BGP Timer Policies at the VRF level, applying these policies directly via the APIC on each respective site is the standard and supported method. 

Session Reset: Yes, a BGP session reset (hard reset) is required for the new timer values to take effect. Because these timers are negotiated during the initial BGP handshake, the existing sessions must be torn down and re-established to renegotiate the new keepalive and hold-down values with the neighbor.

Two things worth underlining from that response: this is a per-site, APIC-side configuration — NDO is not the tool for VRF-level BGP timers today — and a hard reset is mandatory, not optional, for the values to actually apply.

Step-by-Step: Applying the BGP Timer Policy to a VRF via APIC

1. Log into the APIC GUI for the site containing the target VRF (do this per site for stretched VRFs — there's no single-pane push for this from NDO).
2. Navigate to Tenants → <tenant> → Networking → VRFs → <vrf-name>.
3. Under the VRF, go to the BGP Context Per Address Family (create one for each address family in use — IPv4 and/or IPv6).
4. Set the BGP Timers field to a custom BGP Timer Policy, or create a new one:
   • Go to Tenants → <tenant> → Policies → Protocol → BGP → BGP Timers, right-click and Create BGP Timers Policy.
   • Set Keepalive Interval (sec) and Hold Interval (sec) to your target values (keep the standard 1:3 ratio — e.g., 4/12 or 10/30 — most implementations expect hold time to be roughly 3x keepalive).
5. Apply the policy to the VRF's BGP Context Per Address Family and submit.
6. Repeat per site for any stretched VRF — this is not propagated automatically across the Multi-Site domain since it's an APIC-local object, not an NDO-managed one.
7. Confirm the object landed correctly:

   GET https://<apic>/api/node/mo/uni/tn-<tenant>/ctx-<vrf>/bgpCtxAfP-ipv4-ucast.json

Check for the tnBgpTimersPolName attribute pointing to your new policy.

Why the Timers Won't Show Up Until You Reset the Session

This trips people up every time: BGP keepalive and hold timers are exchanged in the OPEN message during session establishment — they're not a live, continuously-enforced parameter that updates on the fly. Changing the policy updates the configuration, but an already-established session keeps running on whatever timers it originally negotiated until it's torn down and rebuilt.

That means after step 7 above, show bgp will show your configured timers, but the negotiated/active timers on any existing session will still be the old values — until you force a reset.

Verifying the Change Actually Took Effect

Don't just trust the APIC MO or the CLI's "configured" line — verify the negotiated value, which is what's actually governing the session.

1. Check the configured value on the border leaf:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

Look for: Configured hold time is 12, keepalive interval is 4 seconds

2. Check the negotiated (active) value in the same output: Look for: Hold time is X, keepalive interval is Y seconds If X/Y still show the old defaults, the session hasn't renegotiated — move to step 3.

3. Force the hard reset:

clear bgp <afi> unicast <peer-ip> vrf <tenant>:<vrf-name>

A soft clear will not renegotiate timers — timers only get exchanged at session establishment, so this has to be a hard clear.

4. Re-check the negotiated value after the reset:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

The negotiated value will be the lower of what each side offers — if your peer's own BGP timer configuration is set higher (or lower) than yours, that governs the final negotiated result, not your configured value alone.

5. Confirm real-world keepalive cadence:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last read"

Run this two or three times a few seconds apart — "Last read" should reset roughly every N seconds, matching your new keepalive interval.

6. Watch for flapping after the change, especially with aggressive timers like 4/12:

show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last reset"
show logging logfile | include BGP

A Word of Caution on Aggressive Timers

Tighter timers (like 4-second keepalive / 12-second hold) improve failure-detection speed, but they raise the risk of false-positive flaps on links with variable latency, high CPU load on the peer device, or congested control-plane paths. Before rolling this out broadly across a Multi-Site fabric, test it on a single non-critical VRF/L3Out first and monitor for a few days.

FAQ

Does NDO support VRF-level BGP timers at all? Not currently. The only BGP timer construct exposed in NDO is at the L3Out node group level (L3Out Node Routing Policy → BGP Node Settings). VRF-level BGP Timer Policies must be applied directly on APIC, per Cisco TAC's confirmation.

Will this survive an NDO template push or resync? Since the VRF-level BGP Timers Policy is applied directly on APIC and not modeled in the NDO template, it should not be overwritten by a standard template redeploy — but any workflow that deletes and recreates the VRF from NDO would remove it. Treat it as an out-of-band configuration and document it clearly for your team.

Do I need to reset the session on both sides of the peering, or just my side? Just your side's session needs a hard clear to force renegotiation — BGP will tear down and rebuild the full session, which inherently re-triggers the handshake and timer exchange on both ends.

What's a safe keepalive/hold ratio to use? Cisco and most vendors expect hold time to be roughly 3x keepalive (e.g., 4/12, 10/30, 20/60). Going below that ratio, or setting a very low hold time on a link with real latency or jitter, increases false-positive flap risk.

Is this the same procedure for a stretched VRF across multiple sites? Yes, but it must be repeated per site — this is an APIC-local object, and NDO doesn't propagate it across the Multi-Site domain automatically. For a stretched VRF, log into each site's APIC and apply the same BGP Timer Policy independently.

Related Reading on Networklearner

• Why BGP Maximum-Paths Wasn't the Problem: Fixing Uneven Port-Channel Traffic on Cisco Nexus and IOS
• Complete Firewalls Port Reference for Cisco ACI, Nexus Dashboard, NDO and NDI
• BGP: Active Open Failed - No Route to Peer
• Weight Attribute - BGP
• BGP Attributes: Well Known vs Optional
• BGP Attribute - AS Prepend
• Cisco ACI Static EPG Configuration – Step-by-Step Deployment Guide
• Complete Steps to Create vPC in Cisco ACI (via APIC GUI)

---

Author note: 

Need help with Cisco ACI, Nexus, data center networking, or network automation?
I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.
Contact me for consulting, troubleshooting, design reviews, and project support.

Complete Firewalls Port Reference for Cisco ACI, Nexus Dashboard, NDO and NDI

 

If you've ever tried to onboard an APIC site into Nexus Dashboard Orchestrator (NDO) or Nexus Dashboard Insights (NDI) only to watch the connection sit at "Not Reachable," the cause is almost always the same: a firewall or ACL between Nexus Dashboard and the ACI fabric is blocking a port nobody told you about.

This guide consolidates every port you need — APIC to leaf/spine, APIC to Nexus Dashboard, NDO-specific ports, NDI-specific ports, and the internal ports a multi-node ND cluster needs between its own nodes — into one reference you can hand straight to your firewall team.

Quick jump:

• Why this trips people up
• Management vs. data network — pick correctly
• Port table: ND cluster ↔ outside world
• Port table: NDO ↔ APIC
• Port table: NDI ↔ APIC / fabric
• Port table: APIC ↔ Leaf/Spine (fabric internal)
• Troubleshooting checklist
• FAQ

---

Why this trips people up

Nexus Dashboard isn't one service — it's a platform that runs NDO, NDI, and NDFC as apps on top of a shared cluster. Each app has its own port requirements on top of the base cluster ports, and each app can reach the fabric through a different interface (management network vs. data network, in-band vs. out-of-band APIC). Miss that distinction and you'll open the right port on the wrong interface — the connection still fails, and it looks identical to a wrong-port problem.

---

Management vs. data network — pick correctly {#management-vs-data-network}

Nexus Dashboard interface	What it's for	Reaches APIC via
Management (mgmt) network	UI access, NTP, DNS, firmware upgrades, Intersight, DC proxy	Can reach APIC OOB or in-band, if routed
Data network	Node-to-node cluster traffic, and all app traffic — NDO, NDI, NDFC	APIC in-band or OOB for NDO; in-band only for NDI

Rule of thumb: if you're only running NDI, or NDI alongside other services, the ND data interface must have IP reachability to the in-band management network of the APIC and every switch in the fabric — NDI pulls switch-level telemetry that OOB simply doesn't carry. NDO is more flexible and can use either OOB or in-band APIC reachability.

---

Port table: ND cluster ↔ outside world {#nd-cluster-ports}

These are baseline ports every Nexus Dashboard cluster needs, regardless of which app (NDO/NDI/NDFC) you run on top.

Also allow outbound to your NTP, DNS, and (if used) TACACS+/LDAP/RADIUS servers, plus internet/proxy access if the cluster needs firmware downloads or Intersight connectivity.

---

Port table: NDO ↔ APIC {#ndo-apic-ports}

NDO's actual data-plane traffic to sites is just the APIC REST API over TCP 443. If you're only building day-0/day-1 policy through NDO (no NDI), this is the shortest port list of the three.

---

Port table: NDI ↔ APIC / fabric {#ndi-apic-ports}

NDI is the most sensitive to the in-band requirement — before onboarding a fabric, confirm the in-band management EPG, bridge domain, and subnet are already built and that every leaf, spine, and APIC has a reachable in-band address. This is the step people skip and then spend hours troubleshooting "site unreachable" errors that are actually EPG/BD issues, not firewall issues.

---

Port table: APIC ↔ Leaf/Spine (fabric internal) {#apic-fabric-ports}

This traffic stays inside the ACI infra VRF and normally isn't firewalled, but it's worth documenting for anyone running APIC connectivity through an external device (e.g., a device sitting between APIC and a remote-leaf pair):

If you're extending remote leaf switches over a routed IPN, also confirm OSPF, DHCP relay, and multicast (PIM Bidir) are permitted, with a minimum MTU of 9150 bytes end to end.

---

Troubleshooting checklist {#troubleshooting}

Work through these in order before opening a TAC case:

1. Ping test first. Confirm basic IP reachability from the ND data interface to the APIC in-band (for NDI) or in-band/OOB (for NDO) address. No ping = no point checking ports yet.
2. Confirm which interface ND is actually using. Nexus Dashboard > Infrastructure > Cluster Configuration shows whether the data or management network is used for fabric connectivity.
3. Verify APIC in-band is actually built (mgmt tenant, in-band BD, in-band EPG, node management addresses) — this is the single most common NDI onboarding blocker, not the firewall itself.
4. Test port 443 directly from an ND node to the APIC IP with curl -vk https://<apic-ip> or openssl s_client -connect <apic-ip>:443.
5. Check for SSL inspection/proxy devices sitting in the path — APIC's certificate handling doesn't play well with transparent SSL interception.
6. Re-check site status after any firewall change — NDO/NDI cache connectivity state and may need a manual "Refresh" or re-registration, not just a rule change.

---

FAQ {#faq}

Does NDO need port 80, or just 443? In current releases, TCP 443 alone is sufficient for APIC REST API communication. Port 80 was referenced in older MSO documentation and is generally not required if 443 is reachable.

Can NDI use the APIC out-of-band address? No. NDI depends on in-band reachability to both APIC and the switches for telemetry collection; OOB-only connectivity will not work for NDI.

Do NDO and NDI need different firewall rules if both run on the same ND cluster? Yes — treat them as separate rule sets. NDO needs 443 to APIC. NDI additionally needs 443 (and sometimes 22) to every switch in the fabric over the in-band network.

What's the minimum MTU I need between Nexus Dashboard and the fabric? 1500 bytes minimum on the ND data interface; higher MTU is supported if your infrastructure already runs jumbo frames.

My rules look correct but the site still shows "Not Reachable." What next? Check for SSL-inspecting firewalls/proxies in the path, confirm the APIC certificate hasn't expired, and verify NTP is in sync across APIC and ND — clock skew alone can break TLS session establishment.

---

Related reading on Networklearner

• ThousandEyes Firewall Requirements: Agent to Dashboard Ports and Rules — same port-table format, useful cross-link since readers researching ACI firewall rules often also manage ThousandEyes agents
• Recovering and Reinitializing a Standby APIC to Active APIC in Cisco ACI
• How to Safely Decommission a Leaf Switch in Cisco ACI — Step-by-Step Guide with Checklist
• Cisco ACI Decommission Only vs Remove vs Secure Remove — Complete Guide with Management IP Behavior
• ACI Leaf Switch Replacement
• Complete Steps to Create vPC in Cisco ACI (via APIC GUI)
• Cisco ACI Static EPG Configuration – Step-by-Step Deployment Guide
• What is a Contract Preferred Group in ACI?
• COOP: Council of Oracle Protocol - Cisco ACI
• Configuring Port Profiles in Cisco ACI (NX-OS Style CLI)
• Cisco ACI Mis-Cabling Protocol (MCP) – Loop Detection Simplified

Cisco ACI Service Graph Management Models Explained

Cisco ACI provides three distinct approaches to manage service graphs, each offering different levels of control and integration:

1. Network Policy Mode (Unmanaged)
   In this mode, ACI configures only the network aspects of the service graph within the fabric. No configuration changes are pushed to the L4-L7 device, making it suitable when device policies are managed externally.

2. Service Policy Mode (Managed)
   Here, ACI not only handles the fabric configuration but also manages VLAN settings on the L4-L7 device. The APIC administrator can directly input device-specific configurations through the APIC interface, ensuring centralized control.

3. Service Manager Mode
   This model allows the firewall or load balancer administrator to define L4-L7 policies. ACI takes care of the fabric and VLAN configurations, while the APIC administrator links these policies with the network policy, enabling a collaborative approach.

Choosing the Right Cisco ACI Service Graph Mode

When designing with Cisco ACI, selecting the right service graph mode depends on your operational needs:

• Dynamic Configuration Needs?
  If firewalls and load balancers must be configured dynamically through APIC, choose Service Policy Mode. If a separate administrator handles device configuration, opt for Network Policy Mode or Service Manager Mode.

• Frequent Commissioning Like Cloud Services?
  For environments where devices are frequently added or removed, Service Policy Mode or Service Manager Mode works best. If services remain static for long periods, Network Policy Mode or Service Manager Mode is more practical.

• Complex Multi-Leg Designs?
  If your design requires multiple interfaces or DMZ configurations, manual service insertion using EPGs and bridge domains may be more convenient than using a service graph.

Bottom Line:
Your choice should align with automation needs, operational flexibility, and design complexity.

Cisco ACI - Node Stateful Vs Stateless reload

 

Aspect	Stateful Reload	Stateless Reload
Definition	Reload where process state is preserved using checkpoints	Reload where process starts fresh without any prior state
State Preservation	Yes – runtime state is saved to Persistent Storage Services (PSS)	No – process is restarted without retaining previous state
Recovery Speed	Faster – resumes from last known state	Slower – requires full reinitialization
System Impact	Minimal – seamless continuation of operations	Higher – may cause temporary disruption or delay
Use Case	Preferred for critical services needing quick recovery	Used when state cannot be preserved or process needs a clean start
Managed By	Persistent Storage Services (PSS)	System Manager
Example Scenario	Restarting a service with session data intact	Replacing a crashed process with a new instance

 

ACI Contracts Components - Contacts Vs Filters Vs Aliases Vs Labels

 In Cisco ACI (Application Centric Infrastructure), Contracts are a key component of the policy model, used to define how endpoints (EPGs) communicate with each other. Within contracts, the terms Labels, Filters, Aliases, and Subjects each play distinct roles. Here's a breakdown of each:

---

🔹 1. Filters

• Purpose: Define the actual traffic (protocols, ports) that is allowed or denied.
• Details:
• Filters are composed of entries specifying Layer 4 information like TCP/UDP ports and protocols.
• They are reusable across multiple contracts.
• Example: A filter might allow TCP traffic on port 80 (HTTP).

---

🔹 2. Subjects

• Purpose: Act as containers within a contract that reference filters and define directionality.
• Details:
• A contract can have multiple subjects.
• Each subject can reference one or more filters.
• You can specify whether the traffic is unidirectional or bidirectional.
• Example: A subject might define that HTTP traffic is allowed from EPG A to EPG B.

---

🔹 3. Aliases

• Purpose: Provide a way to abstract or alias filters for reuse or simplification.
• Details:
• Aliases are less commonly used and are more relevant in complex policy models.
• They can help in referencing filters indirectly, making policy definitions more modular.

---

🔹 4. Labels

• Purpose: Used for categorization and policy enforcement.
• Details:
• Labels can be applied to contracts, EPGs, and other objects.
• They help in grouping and applying policies based on tags.
• Useful in large environments for automation and policy scaling.

---

🧩 How They Work Together in a Contract

• A Contract contains one or more Subjects.
• Each Subject references one or more Filters (or Aliases to filters).
• Labels can be used to tag contracts or EPGs for organizational or policy purposes.

 

Taboo vs vzAny Contract

 

Aspect	Taboo Contract	vzAny Contract
Purpose	Used to explicitly deny certain types of traffic for an EPG	Used to apply contracts to all EPGs within a VRF in a simplified manner
Functionality	Acts as a deny filter for specific traffic (e.g., block port 80 or 23)	Acts as a wildcard to apply contracts to all EPGs in a VRF
Application Scope	Applied to individual EPGs to block traffic	Applied to entire VRF, affecting all EPGs within it
Use Case	Prevent specific traffic types (e.g., cleartext communication)	Enable free intra-VRF communication or many-to-one service consumption
Configuration Complexity	Requires manual filter creation and careful application	Simplifies configuration by automating contract relationships
TCAM Usage	May consume more TCAM entries depending on filter granularity	Optimizes TCAM usage by reducing entries to a single group-level contract
Best Practices	Generally discouraged unless absolutely necessary 1	Recommended for efficient policy management in large-scale environments 2 3
Limitations	Not suitable for inter-EPG communication control	Must follow strict guidelines to avoid unintended traffic leaks 2 3

Recovering and Reinitializing a Standby APIC to active APIC in Cisco ACI

 Recovering and Reinitializing a Standby APIC in Cisco ACI

If you're working with a standby APIC and need to verify access or promote it to an active role, here’s a practical approach to follow.

---

🔐 Step 1: Accessing the Standby APIC

You can attempt to log in using the rescue-user account. This account is specifically designed for emergency access to standby controllers.

• Try using your standard APIC admin password first.
• If the standby APIC is in sync with the cluster, this should work.
• If not, attempt login without a password — some configurations allow password-less access for rescue-user in isolated standby mode.

---

🔄 Step 2: Reinitializing the Standby APIC as Active

If you need to promote a standby APIC (e.g., APIC3) to an active role, it's best to wipe and reconfigure it from scratch. Ensure you have KVM access to the APIC before proceeding.

Run the following commands in sequence:

1. Clean the existing configuration.
2. Trigger the setup script on next boot.
3. Reboot the APIC to begin the reinitialization process.

These commands will:

• acidiag touch clean
• acidiag touch setup
• acidiag reboot

Once rebooted, the APIC will prompt you to configure it as part of the active cluster.

---

🧠 Pro Tip

Always ensure that the APIC you’re reinitializing is physically connected to the fabric and that you have console or KVM access. This process is irreversible and should only be done when you're certain the APIC is no longer needed in standby mode.

 

 Cold Standby APIC – Key Characteristics

• Supported in both single-pod and multi-pod ACI deployments.
• Can be connected to any leaf switch in any pod, restoring edit capabilities in minority scenarios.
• Automatically receives firmware updates to stay in sync with the active APIC cluster.
• During upgrades, once all active APICs are updated, the standby APIC is upgraded automatically.
• Assigned temporary node IDs; a new ID is issued when promoted to active.
• Admin login is disabled on standby APICs.
• Troubleshooting is done via SSH using the rescue-user account.
• During switchover, the replaced active APIC is powered down to avoid dual connectivity.
• Standby APICs do not participate in policy configuration or fabric management.
• Cisco recommends placing standby APICs in the same pod as the active ones they may replace.
• No configuration or credential data is replicated to standby APICs — only rescue-user access is available.

 

L3Out Subnet Scope Options in Cisco ACI

 

L3Out Subnet Scope Options in Cisco ACI

Scope Option	Purpose
Export Route Control Subnet	Advertises specific transit routes out of the ACI fabric. Used for controlling which external routes are exported.
Import Route Control Subnet	Allows specific external routes to be imported into the ACI fabric using BGP or OSPF.
External Subnets for External EPG (Security Import Subnet)	Enables data plane traffic between external EPGs and internal EPGs. Without this, traffic is dropped even if routes are learned.
Shared Route Control Subnet	Used in inter-VRF route leaking. Marks subnets that should be advertised across VRFs.
Shared Security Import Subnet	Similar to External Subnets, but for shared L3Outs. Enables traffic flow between shared external and internal EPGs.
Aggregate Export/Import/Shared Routes	Used to summarize routes, e.g., adding /32 in front of 0.0.0.0/0 for default route aggregation.

---

🧠 Key Notes

• These scope options are critical for controlling route advertisement and enabling secure communication between different parts of the network.
• Cisco ACI uses a contract-based model, so even if routes are advertised, traffic will be dropped unless explicitly allowed by contracts and security prefixes