Showing posts with label Data Center Networking. Show all posts
Showing posts with label Data Center Networking. Show all posts

Sunday, 20 September 2026

Cisco ACI Contract Exceptions vs Subject Exceptions Explained with Examples and Configuration Guide

Introduction

Cisco ACI contracts are used to control communication between Endpoint Groups (EPGs). They provide application-centric security by defining which EPGs can communicate and what traffic is permitted.

In large-scale environments, a single contract may be consumed by many EPGs. However, there are situations where certain EPGs must be excluded from using that contract without creating multiple duplicate contracts.

To solve this challenge, Cisco introduced Contract Exceptions and Subject Exceptions in APIC Release 3.2(1).

These features allow administrators to selectively deny contract participation while maintaining a clean, scalable, and easy-to-manage policy model.


Why Contract Exceptions and Subject Exceptions Are Needed

Consider a production application environment with the following EPGs:

Consumer EPGs:

  • Web-EPG
  • App-EPG
  • Reporting-EPG
  • Test-EPG

Provider EPG:

  • Database-EPG

Contract:

  • DB-Access

Normally all consumer EPGs can access the database through the contract.

Web-EPG → Database-EPG

App-EPG → Database-EPG

Reporting-EPG → Database-EPG

Test-EPG → Database-EPG

Now assume the security team decides that Test-EPG should not access the production database.

Without Contract Exceptions, you would need to create additional contracts and duplicate policies.

With Contract Exceptions, you can simply exclude Test-EPG from participating in the contract.


What is a Contract Exception?

A Contract Exception selectively excludes an EPG from participating in a contract.

The key point is that the exception applies to ALL subjects within the contract.

For example:

Contract Name: DB-Access

Subjects:

  • SQL
  • Backup
  • Monitoring

If Test-EPG is configured as a Contract Exception, it will be denied access to every subject under the contract.

Result:

  • Web-EPG = Allowed
  • App-EPG = Allowed
  • Reporting-EPG = Allowed
  • Test-EPG = Denied

This is useful when an EPG should be completely excluded from the contract.


What is a Subject Exception?

A Subject Exception applies only to a specific subject inside a contract.

Instead of blocking an entire contract, it blocks access only to a particular service.

For example:

Contract Name: Shared-Services

Subjects:

  • HTTP
  • HTTPS
  • SQL
  • DNS

Requirement:

  • HTTP = Allowed
  • HTTPS = Allowed
  • DNS = Allowed
  • SQL = Denied

In this scenario, a Subject Exception is the ideal solution because only the SQL subject needs to be restricted.


Contract Exception vs Subject Exception

Contract Exception affects the entire contract.

Subject Exception affects only a single subject.

Contract Exception is used when an EPG must be excluded completely.

Subject Exception is used when only specific services must be denied.

Contract Exception provides broader control.

Subject Exception provides finer granular control.


Real-World Production Example

Consider a banking environment.

Consumer EPGs:

  • Internet-Web
  • Mobile-App
  • ATM-Services
  • Audit-Tools

Provider EPG:

  • Database-EPG

Contract:

  • Database-Access

Business Requirement:

  • Internet-Web = Allowed
  • Mobile-App = Allowed
  • ATM-Services = Allowed
  • Audit-Tools = Denied

Without Contract Exceptions, multiple contracts would be required.

Database-Access

Database-Access-NoAudit

Database-Access-ATM

Database-Access-Web

This creates unnecessary complexity.

A better design is:

One shared Database-Access contract plus a Contract Exception for Audit-Tools.

This reduces contract sprawl and simplifies management.


Cisco ACI Contract Exception Configuration

Objective

Allow:

  • Web-EPG
  • App-EPG
  • Reporting-EPG

Deny:

  • Test-EPG

For the DB-Access contract.

Step 1: Create the Contract

Navigate to:

Tenant → Contracts → Standard → Create Contract

Configure:

Contract Name: DB-Access

Scope: Tenant

Step 2: Create a Subject

Open the DB-Access contract.

Navigate to:

Subjects → Create Subject

Configure:

Subject Name: SQL

Step 3: Create a Filter

Create a filter with:

Protocol: TCP

Destination Port: 1433

Action: Permit

Attach the filter to the SQL subject.

Step 4: Provide the Contract

Navigate to Database-EPG.

Under Provided Contracts, attach DB-Access.

Step 5: Consume the Contract

Navigate to:

  • Web-EPG
  • App-EPG
  • Reporting-EPG
  • Test-EPG

Attach DB-Access as a Consumed Contract.

At this stage all EPGs can access the database.

Step 6: Create Contract Exception

Navigate to:

Tenant → Contracts → DB-Access → Exceptions

Click Create Exception.

Configure:

Exception Name: Block-Test-EPG

Consumer EPG: Test-EPG

Save and Deploy.

Final Result

  • Web-EPG → Allowed
  • App-EPG → Allowed
  • Reporting-EPG → Allowed
  • Test-EPG → Denied

Because the exception is configured at the contract level, all subjects are affected.


Cisco ACI Subject Exception Configuration

Objective

Allow Web-EPG to access:

  • HTTP
  • HTTPS
  • DNS

Deny access to:

  • SQL

Step 1: Create Contract

Create a contract called Shared-Services.

Step 2: Create Subjects

Create the following subjects:

  • HTTP
  • HTTPS
  • SQL
  • DNS

Step 3: Attach Filters

HTTP Subject:

TCP Port 80

Permit

HTTPS Subject:

TCP Port 443

Permit

SQL Subject:

TCP Port 1433

Permit

DNS Subject:

UDP Port 53

Permit

Step 4: Create Subject Exception

Navigate to:

Shared-Services → Subject SQL → Exceptions

Click Create Exception.

Configure:

Exception Name: Web-No-SQL

EPG: Web-EPG

Save and Deploy.

Final Result

For Web-EPG:

  • HTTP → Allowed
  • HTTPS → Allowed
  • DNS → Allowed
  • SQL → Denied

Other EPGs continue to access SQL if no Subject Exception exists for them.


How Packet Processing Works

When traffic is initiated:

Step 1: Cisco ACI identifies the source EPG.

Step 2: Cisco ACI identifies the destination EPG.

Step 3: Contract lookup is performed.

Step 4: Subject filters are evaluated.

Step 5: Contract and Subject Exceptions are checked.

Step 6: Traffic is either permitted or denied.

This enables very granular policy enforcement within the ACI fabric.


Logging and Visibility Features

Contract Exceptions and Subject Exceptions support:

Labels

Useful for policy identification.

Examples:

  • Production-SQL
  • PCI-Restricted
  • Database-Security

Counters

Used to track:

  • Permit Hits
  • Deny Hits
  • Bytes
  • Packets

Permit and Deny Logging

Helpful during troubleshooting and audits.

Administrators can determine whether traffic was blocked due to:

  • Contract Exception
  • Subject Exception
  • Deny Filter
  • Missing Contract

Troubleshooting Tips

Verify zoning rules using:

show zoning-rule scope tenant

Monitor contract counters from:

Tenant → Operations → Counters → Contracts

Review:

  • Permit Hits
  • Deny Hits
  • Packet Counts
  • Byte Counts

Check Events and Faults under:

Operations → Events

Operations → Faults

This helps quickly identify policy-related connectivity issues.


Design Best Practices

Use Contract Exceptions when:

  • Development EPGs must be blocked completely.
  • Test EPGs must be isolated.
  • Audit environments should not reach production systems.
  • Temporary projects require full exclusion.

Use Subject Exceptions when:

  • HTTP should be allowed but SQL denied.
  • HTTPS should be allowed but SSH denied.
  • DNS should be allowed but ICMP denied.

Avoid creating multiple duplicate contracts.

Instead of:

Web-To-DB

App-To-DB

ATM-To-DB

Audit-To-DB

Use:

Shared Database Contract

Plus

Contract Exception or Subject Exception

This creates a simpler and more scalable design.


Interview Questions

Q1. What is the difference between Contract Exception and Subject Exception?

Contract Exception applies to the entire contract and affects all subjects. Subject Exception applies only to a specific subject within the contract.

Q2 Can Contract Exceptions be configured for both Providers and Consumers?

Yes. Contract Exceptions can exclude both provider and consumer EPGs.

Q3. When should Subject Exceptions be used?

When only specific application services need to be denied while allowing other services within the same contract.

Q4. Which APIC release introduced Contract and Subject Exceptions?

APIC Release 3.2(1).

Q5. What is the main benefit of Contract Exceptions?

They reduce contract sprawl and simplify policy management.


Key Takeaways

  • Contract Exceptions apply to the entire contract.
  • Subject Exceptions apply to a specific subject only.
  • Both features were introduced in APIC 3.2(1).
  • They help implement granular micro-segmentation.
  • They reduce the need for duplicate contracts.
  • Logging, counters, labels, and deny/permit monitoring are fully supported.
  • They improve scalability and operational simplicity in large Cisco ACI deployments.

Conclusion

Contract Exceptions and Subject Exceptions are powerful Cisco ACI features that help architects implement flexible micro-segmentation policies without increasing configuration complexity. Contract Exceptions are ideal when an EPG must be excluded from an entire contract, while Subject Exceptions are best suited for restricting access to specific services within a contract. By using these features effectively, organizations can simplify policy management, reduce contract sprawl, improve security, and maintain a cleaner ACI policy model across the fabric.

Recommended for: CCNP Data Center, CCIE Data Center, ACI Administrators, Data Center Engineers, Network Architects, and anyone designing secure Cisco ACI environments.


Summary


Related Links

  • Cisco ACI MoQuery – Advanced Commands for Day-to-Day Operations
    Useful for verifying contracts, providers/consumers, subjects, filters, and Taboo contracts.
    Read the MoQuery guide
  • What is a Contract Preferred Group in ACI?
    Related to ACI contract policy and EPG communication within a VRF.
    Read the Contract Preferred Group guide
  • Cisco ACI Explained: Concepts, Learning Prerequisites, Benefits, and Limitations
    Provides a broader introduction to ACI contracts, EPGs, filters, and the deny-by-default model.
    Read the Cisco ACI guide
  • Top 10 Cisco ACI Multiple Choice Questions (MCQs)
    Useful for reinforcing contract scope, filter direction, consumer/provider relationships, and vzAny.
    Read the ACI MCQs
  • L3Out Subnet Scope Options in Cisco ACI
    Useful as a related policy/security topic, particularly for understanding external EPG security import subnets and contract-based traffic control.
    Read the L3Out Subnet Scope guide

  • Saturday, 12 September 2026

    Cisco ACI: How Overlapping L3Out Subnets Silently Hijack Contracts

    If you run multiple L3Outs in the same VRF on Cisco ACI, there's a classification behavior that catches a lot of engineers off guard: external EPG classification is done per-VRF, not per-L3Out. That single fact can cause traffic from one L3Out to get tagged — and policed — as if it came from a completely different one.

    I ran into this in the lab and wanted to break it down in plain terms, because the failure mode is sneaky: everything can look and even work fine, right up until it doesn't.

    The Lab Topology

    Two leaf switches, one VRF, two L3Outs:

    • Leaf-101 — OSPF L3Out on interface eth1/22 (192.168.48.1/24), external EPG pcTag 16387
    • Leaf-102 — EIGRP L3Out on an SVI (192.168.47.1/24), external EPG pcTag 16386
    • An App EPG (pcTag 49153) sitting on Leaf-101, with an endpoint at 192.168.1.17/24

    OSPF is advertising a host route, 192.168.9.6/32. EIGRP is advertising 192.168.9.1/32. On the EIGRP L3Out, the admin has flagged 192.168.9.0/24 as "External Subnet for External EPG" — a common shortcut to catch a whole range of addresses under one external EPG instead of listing every host route.

    Here's the setup visually:


    What ACI Actually Does With That /24

    This is the part that trips people up. In ACI, an external (L3Out) EPG isn't classified by which physical L3Out or interface the packet arrived on — it's classified by a longest-prefix-match (LPM) lookup against a single table shared by the whole VRF. Every leaf switch running that VRF gets a copy of the same prefix-to-pcTag table.

    So the moment 192.168.9.0/24 gets programmed as "external subnet for external EPG" under the EIGRP external EPG, that entry doesn't just live on Leaf-102. It gets pushed to every leaf in the VRF, including Leaf-101 — the one that only knows about OSPF.

    Where It Goes Wrong

    Before any contract touches the EIGRP external EPG, Leaf-101 only has a generic 0.0.0.0/0 catch-all entry in its prefix table. Traffic from 192.168.9.6 (the OSPF route) matches that default and gets classified correctly as OSPF's external EPG (16387). No drama.

    Add just one contract anywhere that references the EIGRP external EPG — even a contract to something entirely unrelated to Leaf-101 — and ACI pushes the 192.168.9.0/24 entry into Leaf-101's classification table too.

    Now watch what happens to traffic sourced from 192.168.9.6 hitting Leaf-101 via the OSPF L3Out:

    • 192.168.9.6 matches both the OSPF default route (0.0.0.0/0) and the new EIGRP entry (192.168.9.0/24)
    • LPM always wins with the more specific prefix
    • /24 beats /0, so the packet gets tagged with pcTag 16386 (EIGRP) — not 16387 (OSPF), even though it arrived over the OSPF L3Out and has nothing to do with EIGRP

    The switch isn't misbehaving. It's doing exactly what LPM tables are supposed to do. The problem is that two unrelated EPGs ended up sharing address space in a VRF-wide table.

    Why It Doesn't Always Blow Up Immediately

    In my test, a ping from 192.168.9.6 to the App EPG endpoint still succeeded — even with no contract explicitly permitting OSPF-to-App traffic. Why? Because the traffic got misclassified as EIGRP (16386), and a contract already existed between the EIGRP external EPG and the App EPG. Wrong tag, right permission — so it slipped through.

    That's the dangerous part: the traffic looks fine on the surface, but it's being permitted for the wrong reason. And the only reason it worked at all in this lab comes down to one coincidence — which is worth unpacking on its own.

    Scenario 2: Add a Third Leaf and the "Lucky Pass" Disappears

    The reason the misclassified ping went through in the two-leaf lab is that the App EPG endpoint happened to live on the same leaf (Leaf-101) where the misclassification occurred. ACI doesn't blast every zoning-rule to every leaf in the fabric — a rule only gets programmed onto the leaves where it's actually needed, based on where the relevant EPGs have local significance (an endpoint, a BD, or an L3Out). Leaf-101 already needed the pcTag 16386 ↔ 49153 rule locally, since it happened to host the App EPG endpoint too — so the misdirected OSPF traffic rode along on a rule that was only there for an unrelated reason.

    Now change one thing: put the App EPG endpoint on a separate, third leaf (Leaf-103) instead of Leaf-101. The traffic path looks like this:





    Here's what happens:

    • Traffic from 192.168.9.6 still enters through Leaf-101 (the OSPF border leaf) and still gets misclassified as pcTag 16386, for the exact same LPM reason as before.
    • Policy enforcement in this VRF happens on ingress, so the permit/deny decision for that traffic is made right there on Leaf-101 — before it ever reaches Leaf-103.
    • The zoning-rule for pcTag 16386 ↔ 49153 exists on Leaf-103 (which hosts the App EPG endpoint) and on Leaf-102 (which owns the EIGRP L3Out and enforces policy for return traffic heading back out). Neither of those is Leaf-101.
    • Leaf-101 has no local reason to carry that specific rule, so the misclassified traffic hits the implicit deny at the ingress leaf and is silently dropped — never even reaching Leaf-103.

    So the exact same misclassification bug produces two different outcomes purely based on physical placement of the endpoint: a false "it works" when the endpoint and the misclassifying leaf are the same box, and a silent drop once you scale out to more leaves — which is the normal, realistic case in production. That's what makes this bug so easy to miss in a small lab and so painful to chase down later in a real fabric.

    How to Actually Confirm This Is Happening

    Three places to check on the leaf CLI:

    1. show zoning-rule scope <vrf-vnid> — shows the actual permit/deny rules per pcTag pair (ACI's ACL equivalent).
    2. show system internal policy-mgr prefix — shows what prefixes are currently mapped to which pcTag on that specific leaf. This is where you'll catch the 192.168.9.0/24 → 16386 entry showing up on a leaf that shouldn't care about EIGRP at all.
    3. ELAM (Embedded Logic Analyzer Module) — trigger a capture on the source IP and check the sclass field in the report. If it shows the EIGRP pcTag for OSPF-sourced traffic, that's your smoking gun.

    The Takeaway

    Don't let external subnets marked "External Subnet for External EPG" overlap across L3Outs that share a VRF. Even if the L3Outs are on completely different leaves, different routing protocols, and different interfaces, the classification table doesn't care — it's VRF-wide.

    A few practical guardrails:

    • Keep external subnet ranges assigned per L3Out non-overlapping wherever possible.
    • Avoid broad "catch-all" subnets (like a /24) on an external EPG unless you're certain no other L3Out in the same VRF will ever advertise something inside that range.
    • Before adding a new contract involving an external EPG, check show system internal policy-mgr prefix on the other leaves in the VRF to see what it's about to push fabric-wide.
    • If a ping or flow "just works," don't assume it worked through the contract you think it did — verify the pcTag with ELAM if something feels off.
    • Don't trust a passing test in a small lab as proof the design is safe — a "lucky pass" caused by endpoint placement on a single leaf can mask a bug that fails as soon as you scale to more leaves.

    Related Reading

    Sunday, 23 August 2026

    CI/CD Pipelines for Cisco ACI: Automate, Validate and Deploy Network Changes with Confidence

     Modern data centers are becoming increasingly software-driven. Network engineers are no longer limited to configuring individual switches through the CLI; instead, infrastructure can be represented as code, stored in version-control systems, tested automatically, and deployed through controlled pipelines.

    This is where NetDevOps and CI/CD pipelines become highly relevant to Cisco ACI environments.

    Cisco ACI already provides a policy-driven and API-based approach to data center networking through APIC. By combining these capabilities with Git, automation tools, testing frameworks, and CI/CD platforms, network teams can create a repeatable process for moving an ACI configuration change from development to production.

    The objective is simple:

    Design → Commit → Validate → Test → Approve → Deploy → Verify

    The source material describes the same fundamental CI/CD idea: configuration changes are stored in a repository, validated through automated stages, and eventually deployed instead of being applied directly to the production network.


    What Is NetDevOps?

    NetDevOps applies DevOps principles to network engineering.

    Traditional network operations often look like this:

    Requirement → Engineer logs in → CLI/API change → Verification

    Although this approach can work, it becomes difficult to maintain consistency when hundreds or thousands of configuration changes are performed across a large environment.

    A NetDevOps approach introduces:

    • Version control
    • Automation
    • Automated testing
    • Peer review
    • Repeatable deployments
    • Change tracking
    • Rollback capability
    • Continuous validation

    For Cisco ACI, this means that objects such as:

    • Tenants
    • VRFs
    • Bridge Domains
    • Application Profiles
    • EPGs
    • Contracts
    • Filters
    • L3Outs
    • Interface policies
    • Access policies

    can be managed through an automated and controlled workflow.


    Why Use CI/CD for Cisco ACI?

    Cisco ACI is already designed around centralized policy management. However, manually creating and modifying policies through the APIC GUI or CLI can still introduce operational risks.

    A CI/CD workflow adds another layer of control around those changes.

    1. Version Control

    ACI configuration definitions can be maintained in Git.

    This provides a history of:

    • Who changed the configuration
    • What was changed
    • When it was changed
    • Which version was deployed

    This becomes particularly useful during troubleshooting or rollback.

    2. Automated Validation

    Before a change reaches production, the pipeline can validate:

    • Configuration syntax
    • Required parameters
    • Naming standards
    • IP addressing
    • Policy dependencies
    • Configuration structure
    • Organizational standards

    3. Automated Testing

    The configuration can be deployed to a test or non-production environment before production deployment.

    For example, a pipeline could verify:

    Tenant → VRF → BD → EPG → Contract → Connectivity

    4. Controlled Production Deployment

    Only changes that successfully pass the required validation and approval stages should be promoted to the production ACI fabric.

    5. Better Auditability

    Every pipeline execution can provide an audit trail showing:

    Commit → Test Result → Approval → Deployment → Verification

    This is significantly easier to review than a collection of manually executed CLI commands.


    Understanding the CI/CD Pipeline for Cisco ACI

    A typical ACI CI/CD pipeline can be divided into five major stages:

    Source → Build → Test → Deploy → Verify

    For Cisco ACI, we can extend this model by adding explicit validation and post-deployment verification.


    Stage 1: Source Repository

    The process starts when a network engineer creates or modifies an ACI configuration definition.

    Instead of immediately changing APIC, the configuration is stored in a source-control repository.

    For example:

    ACI-Configuration/
    │
    ├── tenants/
    │   ├── tenant-web.yaml
    │   └── tenant-app.yaml
    │
    ├── vrfs/
    │   └── vrf-production.yaml
    │
    ├── bridge-domains/
    │   └── bd-web.yaml
    │
    ├── epgs/
    │   └── epg-web.yaml
    │
    └── contracts/
        └── contract-web-to-app.yaml

    A network engineer can create a feature branch, make the required change, and submit it for review.

    For example:

    main
      |
      +---- feature/new-web-epg
                  |
                  +---- configuration change
                  |
                  +---- validation
                  |
                  +---- pull request

    This creates a much more controlled change-management process.


    Stage 2: Build and Configuration Validation

    Once a change is committed, the CI/CD pipeline can automatically start.

    The build stage does not necessarily mean compiling software as it would in application development.

    For network automation, the build stage can mean:

    • Loading configuration files
    • Validating YAML or JSON
    • Checking required fields
    • Validating configuration structure
    • Checking dependencies
    • Running policy checks
    • Preparing the deployment package

    For example, a pipeline could detect:

    Tenant: PROD
    VRF: PROD_VRF
    BD: WEB_BD
    Subnet: 10.10.10.0/24
    Gateway: 10.10.10.1

    and verify that the values comply with the organization's standards.

    A failed validation should stop the pipeline before the configuration reaches the ACI fabric.


    Stage 3: Test the ACI Configuration

    Testing is one of the most important parts of the pipeline.

    The objective is not simply to determine whether the configuration can be pushed to APIC.

    The more important question is:

    Does the configuration produce the expected network behavior?

    Depending on the environment, automated tests could verify:

    Tenant Validation

    Is the expected tenant present?

    VRF Validation

    Does the correct VRF exist?

    Bridge Domain Validation

    Is the BD associated with the correct VRF?

    EPG Validation

    Is the EPG associated with the expected BD?

    Contract Validation

    Are the expected contracts and filters configured?

    Connectivity Validation

    Can the required endpoints communicate?

    Policy Validation

    Does the resulting configuration comply with the organization's security and network policies?

    The source document similarly describes using a non-production testing environment to validate network configuration before production deployment.


    Stage 4: Approval and Production Deployment

    After successful testing, the change can move toward production.

    There are several possible deployment models.

    Manual Approval

    The pipeline pauses and asks an engineer to approve the production change.

    Test Passed
         ↓
    Change Review
         ↓
    Engineer Approval
         ↓
    Production Deployment

    This is often a good starting point for production ACI environments.

    Automated Promotion

    More mature environments can automatically promote a change when all required checks pass.

    Commit
      ↓
    Build
      ↓
    Test
      ↓
    Security Check
      ↓
    Approval Policy
      ↓
    Production

    For production ACI, I recommend beginning with human approval and gradually introducing controlled automation.


    Stage 5: Deploy to Cisco ACI

    The pipeline can communicate with APIC using appropriate automation interfaces.

    Possible approaches include:

    • APIC REST API
    • Python-based automation
    • ACI SDK/tooling
    • Ansible
    • Terraform where appropriate
    • Custom automation scripts

    A simplified workflow could look like:

    Git Repository
          ↓
    CI/CD Server
          ↓
    Validation
          ↓
    Automation Engine
          ↓
    Cisco APIC
          ↓
    ACI Leaf/Spine Fabric

    The key advantage is that the engineer does not need to manually repeat the same configuration process for every deployment.


    Stage 6: Post-Deployment Verification

    Deployment should not be considered the end of the pipeline.

    A good network CI/CD pipeline should verify that the intended state actually exists after deployment.

    For example:

    Deploy
      ↓
    Check APIC Response
      ↓
    Check Object State
      ↓
    Check ACI Faults
      ↓
    Check EPG/Contract State
      ↓
    Check Connectivity
      ↓
    Success / Rollback

    This is especially important in ACI because a configuration can be accepted by the controller while the resulting operational state still requires validation.


    A Practical Cisco ACI CI/CD Example

    Imagine an application team requests a new application environment.

    The requirement is:

    Application: Customer-App
    
    Tenant: PROD
    VRF: PROD-VRF
    BD: CUSTOMER-BD
    EPG: CUSTOMER-APP
    Contract: APP-TO-DB

    Instead of manually creating every object through APIC, the engineer creates the desired configuration in the repository.

    The pipeline then performs the following:

    Step 1 — Commit

    The engineer commits the configuration.

    Step 2 — Build

    The pipeline checks the configuration structure.

    Step 3 — Validate

    It checks naming conventions, required fields, and dependencies.

    Step 4 — Test

    The configuration is tested against an ACI test environment or appropriate validation framework.

    Step 5 — Review

    A network engineer reviews the proposed change.

    Step 6 — Deploy

    The approved configuration is pushed through the automation layer to APIC.

    Step 7 — Verify

    The pipeline checks:

    • Tenant
    • VRF
    • BD
    • EPG
    • Contract
    • Faults
    • Connectivity

    Step 8 — Record

    The pipeline records the deployment result.

    This creates a repeatable workflow instead of a one-time manual change.


    Tools Commonly Used in Network CI/CD

    For Cisco ACI environments, these tools can be mapped into a broader workflow.

    FunctionExample Tools
    Source ControlGit, GitLab, GitHub
    PipelineGitLab CI/CD, Jenkins, GitHub Actions
    AutomationAnsible, Python, Terraform
    ACI InterfaceAPIC REST API, ACI SDK/tooling
    TestingpyATS, custom API tests
    Lab/SimulationCisco Modeling Labs
    MonitoringExisting enterprise monitoring platforms
    Configuration FormatYAML, JSON
    ReviewGit Pull/Merge Requests

    The exact combination depends on the organization's architecture and automation standards.


    Git and Cisco ACI: Why Version Control Matters

    One of the biggest changes when moving from traditional networking to NetDevOps is treating configuration as a managed artifact.

    Consider a manual change:

    Engineer → APIC → Configuration Change

    Later someone asks:

    "What exactly changed last Tuesday?"

    The answer may require searching audit logs and controller history.

    With Git:

    Engineer
       ↓
    Git Commit
       ↓
    Review
       ↓
    Pipeline
       ↓
    APIC

    The configuration change becomes part of a structured history.

    This makes troubleshooting and rollback easier.


    CI/CD Does Not Mean "Push Everything Automatically"

    This is an important point for network engineers.

    CI/CD does not mean that every configuration change should automatically reach production.

    A mature network pipeline should include appropriate controls.

    For example:

    Developer
       ↓
    Git Commit
       ↓
    Automated Validation
       ↓
    Automated Testing
       ↓
    Security / Compliance Check
       ↓
    Peer Review
       ↓
    Human Approval
       ↓
    Production
       ↓
    Verification

    The amount of automation can increase as confidence in the pipeline increases.


    What Happens When a Test Fails?

    A good pipeline should stop.

    For example:

    Git Commit
        ↓
    Build
        ↓
    Validation
        ↓
    Test
        ↓
    ❌ FAILED
        ↓
    Stop Pipeline
        ↓
    Notify Engineer

    The engineer can correct the configuration and submit another change.

    This is much safer than discovering the problem after deployment to the production fabric.


    Rollback Strategy for Cisco ACI

    Rollback should be considered before automation is introduced.

    A version-controlled workflow makes it easier to identify the previous known-good configuration.

    A simplified process is:

    Current Version
          ↓
    New Configuration
          ↓
    Deployment
          ↓
    Verification
          ↓
    Failure?
       ↙       ↘
     Yes        No
     ↓           ↓
    Rollback    Continue

    The exact rollback mechanism should be designed carefully according to the ACI objects involved and the organization's change-management procedures.


    Benefits of CI/CD for Cisco ACI Engineers

    Faster Changes

    Repeated configuration tasks can be automated.

    Fewer Manual Errors

    Automated validation reduces common configuration mistakes.

    Consistency

    The same process can be applied repeatedly.

    Better Collaboration

    Network and application teams can work with a common version-control workflow.

    Easier Auditing

    Configuration changes are associated with commits, reviews, pipeline results, and deployment records.

    Safer Production Changes

    Changes can be validated before reaching the production fabric.

    Repeatability

    The same desired configuration can be deployed consistently across environments where appropriate.


    Challenges You Should Consider

    CI/CD is powerful, but it is not a magic solution.

    1. Poorly Designed Automation

    Automating a bad process simply makes the bad process faster.

    2. Insufficient Testing

    If tests do not represent real network behavior, a successful pipeline does not guarantee a successful production deployment.

    3. Secrets Management

    API credentials, tokens, and other sensitive information must be handled securely.

    4. Pipeline Permissions

    The automation account should have only the permissions required for its task.

    5. Production Safety

    High-impact changes should have appropriate approvals and safeguards.

    6. Lack of Rollback Planning

    Every production deployment should have a defined recovery strategy.


    ACI CI/CD Maturity Model

    A network team does not have to automate everything on day one.

    A practical progression is:

    Level 1 — Version Control

    Store ACI configurations in Git.

    Level 2 — Automated Validation

    Validate every configuration change.

    Level 3 — Automated Testing

    Test changes against a non-production environment.

    Level 4 — Controlled Deployment

    Deploy approved changes automatically.

    Level 5 — Continuous Verification

    Automatically verify the resulting network state.

    Level 6 — Advanced Network Automation

    Introduce more sophisticated orchestration, compliance, and intent-based workflows.

    This gradual approach allows network engineers to build confidence without immediately handing complete production control to an automation pipeline.


    NetDevOps Changes the Role of the Network Engineer

    CI/CD does not eliminate the need for network engineers.

    Instead, it changes where their time is spent.

    Traditional model:

    Configure
    Troubleshoot
    Repeat

    Modern model:

    Design
        ↓
    Define Desired State
        ↓
    Automate
        ↓
    Test
        ↓
    Deploy
        ↓
    Observe
        ↓
    Improve

    Network engineers therefore benefit from learning skills beyond traditional CLI configuration.

    Some particularly valuable skills include:

    • Git
    • Python
    • REST APIs
    • YAML/JSON
    • Ansible
    • Terraform
    • CI/CD concepts
    • Automated testing
    • Infrastructure as Code
    • Network programmability

    For Cisco ACI engineers, this combination can be especially powerful because ACI already provides a centralized controller and programmable interfaces.

    Summary





    Final Thoughts

    Cisco ACI provides a strong foundation for policy-driven and automated data center networking. CI/CD and NetDevOps can extend that foundation by introducing version control, automated validation, testing, controlled deployment, and continuous verification.

    The biggest mindset change is to stop thinking of a network change as simply:

    "A command I need to execute."

    Instead, think of it as:

    "A configuration change that should pass through a controlled lifecycle."

    A mature Cisco ACI CI/CD workflow can therefore look like:

    Design → Git → Validate → Test → Review → Deploy → Verify → Monitor

    The goal is not to automate every possible task.

    The goal is to make network changes repeatable, testable, auditable, and safer.

    For network engineers who already work with Cisco ACI, learning CI/CD and NetDevOps is a natural next step toward modern network automation.



    Related Articles from Netterrene

    Cisco ACI Explained: Concepts, Learning Prerequisites, Benefits, and Limitations
    Read the Cisco ACI fundamentals guide

    AI Planning Strategies for Cisco ACI Engineers
    Read the ACI AI planning guide

    How AI Agents Actually "Touch" Your Cisco ACI Fabric
    Read the ACI automation and API guide

    How AI Memory Can Revolutionize Cisco ACI Operations
    Read the ACI AI memory guide

    Multi-Agent AI Systems Explained for Cisco ACI Engineers
    Read the ACI multi-agent guide

    Networklearner – Cisco ACI, Network Automation and AI Articles
    Visit Netterrene


    Need Help With Cisco ACI or Network Automation?

    Need help with Cisco ACI, Nexus, data center networking, or network automation?

    I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.

    Contact me for consulting, troubleshooting, design reviews, and project support free of cost.


    Thursday, 20 August 2026

    How to Permanently Decommission a Leaf Switch in Cisco ACI Fabric: A Complete Guide for Network Engineers

     ACI leaf switch is more than simply unplugging cables and removing the hardware. The switch can be referenced throughout the fabric by policies, vPC configurations, management settings, and routing components. If these dependencies are not removed correctly, the fabric may continue to generate faults and warnings long after the switch has been taken out of service.

    This guide walks through a clean and permanent method for removing a leaf switch from an ACI fabric while ensuring the environment remains stable and fault-free.

    When Would You Need to Decommission a Leaf Switch?

    Network engineers commonly decommission leaf switches during:

    • Hardware refresh projects
    • Data center migrations
    • Fabric redesign initiatives
    • Capacity optimization exercises
    • RMA and hardware replacement activities
    • Retirement of unused infrastructure

    No matter the reason, following a structured process helps prevent operational issues later.


    Why Proper Decommissioning Matters

    Before removing a leaf switch, it is important to understand that the node may still be referenced by:

    • vPC configurations
    • Endpoint Group (EPG) static bindings
    • L3Out configurations
    • Interface policies
    • Management connectivity settings
    • Maintenance groups and policy groups

    Leaving these references behind can generate faults and impact fabric health even after the physical switch has been disconnected.

    Step 1: Remove vPC Dependencies

    If the leaf switch is part of a vPC pair, begin by removing the explicit vPC protection group associated with the node.

    This is one of the most frequently missed steps during decommissioning. If the vPC relationship remains configured, the surviving peer may continue looking for its partner and generate unnecessary faults within the fabric.

    Always verify that all vPC references have been removed before proceeding.

    Step 2: Remove All References to the Leaf Switch

    Before decommissioning the node, review the fabric and remove every configuration object that references the switch.

    Static Path Bindings

    Check all Endpoint Groups (EPGs) and remove any static path bindings associated with the leaf switch.

    L3Out Configuration

    Review:

    • Node Profiles
    • Interface Profiles

    Remove any references to the target node.

    Interface Policies

    Validate and clean up:

    • Interface Selectors
    • Interface Profiles
    • Leaf Switch Profiles

    Management Configuration

    Verify and remove:

    • Out-of-Band (OOB) Management Addresses
    • In-Band (INB) Management Addresses

    Policy Group Membership

    Ensure the node is removed from:

    • Leaf Policy Groups
    • Maintenance Groups
    • Operational Groups

    Cleaning these dependencies beforehand helps ensure a smooth removal process and reduces troubleshooting efforts afterward.

    Step 3: Decommission the Node from APIC

    After confirming that all dependencies have been removed, navigate to:

    Fabric → Inventory → Fabric Membership

    Select the leaf switch you want to retire and choose the Decommission option.

    When prompted, select:

    Remove from Controller

    This is the permanent decommission option.

    When selected, APIC removes:

    • Node ID association
    • Serial number registration
    • Fabric membership information

    After the process completes, the switch is no longer considered an active member of the ACI fabric.

    Step 4: Verify Successful Removal

    Allow a few minutes for APIC convergence and database updates.

    Once convergence is complete, verify the following:

    • The node no longer appears in Fabric Membership
    • The serial number has been removed from APIC inventory
    • No active faults are associated with the node
    • No policies reference the decommissioned switch

    This validation step provides confidence that the node has been successfully removed from the fabric.

    Step 5: Clean the Switch Configuration

    Although the switch has been removed from APIC, the physical device may still retain its ACI identity.

    Connect to the switch console and run:

    setup-clean-config.sh

    After the cleanup process completes, reload the switch:

    reload

    This removes residual ACI configuration and fabric identity information from the hardware.

    Performing this cleanup is highly recommended if the switch will be:

    • Added to another ACI fabric
    • Used in a lab environment
    • Returned to inventory
    • Repurposed for another project

    Step 6: Physically Disconnect the Switch

    The final step is to remove all physical connections.

    Disconnect:

    • Uplinks
    • Downlinks
    • Management cables
    • Power connections (when approved by local procedures)

    Physical removal should always occur after the logical decommissioning process is fully completed.

    Many engineers make the mistake of disconnecting the switch first, which can complicate troubleshooting and validation activities.

    A good rule to remember is:

    "Logically remove first, physically remove last."

    Common Mistakes to Avoid

    Skipping vPC Cleanup

    This can cause vPC-related faults on the peer switch.

    Forgetting Static Path Bindings

    Orphaned EPG configurations can remain in the fabric.

    Leaving L3Out References Behind

    Routing policies referencing decommissioned nodes can cause operational issues.

    Not Selecting "Remove from Controller"

    The switch may continue to appear in the fabric inventory.

    Skipping Switch Cleanup

    The device may retain old fabric identity information and create issues when reused.

    Disconnecting Cables Too Early

    This makes verification and troubleshooting more difficult.

    Best Practice Checklist

    Before retiring a Cisco ACI leaf switch, confirm the following:

    ✅ vPC protection groups removed

    ✅ Static path bindings cleaned

    ✅ L3Out references removed

    ✅ Interface profiles cleaned

    ✅ OOB and INB management addresses removed

    ✅ Policy groups updated

    ✅ Node decommissioned from APIC

    ✅ Serial number removed from Fabric Membership

    ✅ Switch cleaned using setup-clean-config.sh

    ✅ Physical cables disconnected last

    Conclusion

    A clean Cisco ACI leaf switch decommission requires more than simply removing hardware from the rack. By systematically removing policy references, cleaning vPC dependencies, decommissioning the node from APIC, clearing the switch configuration, and finally disconnecting physical cabling, you can avoid unnecessary faults and maintain a healthy fabric.

    Following this approach will help network engineers perform leaf switch retirements confidently while ensuring operational consistency across the data center environment.

    Related Networking Articles

    Continue learning Cisco ACI and Data Center Networking concepts through Netterrene:

    • https://netterrene.blogspot.com/
    • https://netterrene.blogspot.com/search/label/Cisco%20ACI
    • https://netterrene.blogspot.com/search/label/Data%20Center
    • https://netterrene.blogspot.com/search/label/Cisco
    • https://netterrene.blogspot.com/search/label/Networking

    For more Cisco ACI troubleshooting guides, best practices, and real-world operational lessons, visit:

    https://netterrene.blogspot.com/

    Wednesday, 5 August 2026

    How AI Memory Can Revolutionize Cisco ACI Operations

    Cisco ACI has transformed data center networking by introducing policy-based automation, centralized management, and application-centric networking. However, as enterprise fabrics grow, troubleshooting becomes increasingly challenging.

    Imagine receiving multiple faults from different leaf switches after a maintenance window. Instead of manually checking APIC health, interface counters, contracts, endpoint learning, and event logs one by one, an AI assistant could analyze everything in seconds.

    But how can an AI assistant remember previous troubleshooting steps, understand your fabric, and retrieve your organization's deployment standards?

    The answer lies in AI Memory.

    Just like an experienced Cisco ACI engineer relies on operational knowledge, design documents, and previous troubleshooting experience, AI agents use different types of memory to provide intelligent assistance.

    In this article, we'll explore how AI memory works using real Cisco ACI scenarios.


    Why Memory Matters in Cisco ACI

    Suppose your production ACI fabric consists of:

    • Three APIC controllers
    • Two Spine switches
    • Twenty Leaf switches
    • Hundreds of EPGs
    • Multiple VRFs
    • External L3Out connections
    • VMware VMM integration

    Now imagine that an application suddenly loses connectivity.

    Without memory, an AI assistant would ask the same questions every time:

    • Which tenant is affected?
    • Which EPGs are involved?
    • What contracts exist?
    • Which leaf switch hosts the endpoint?
    • Was any policy recently modified?

    An experienced engineer already remembers much of this context. Likewise, AI memory allows an intelligent assistant to retain relevant information, retrieve documentation, and personalize future troubleshooting.


    The Three Types of AI Memory

    Think of AI memory as the way a senior Cisco ACI architect organizes information.

    AI MemoryCisco ACI Example
    Working MemoryCurrent APIC fault investigation
    Long-Term MemoryACI design documents and runbooks
    Episodic MemoryPrevious incidents and administrator preferences

    Together, these memory types help AI solve problems faster and more accurately.


    Working Memory – The Current Troubleshooting Session

    Working memory contains information that the AI agent is actively processing during the current investigation. It is temporary and limited to the ongoing session.

    Cisco ACI Example

    An engineer asks:

    "Why are endpoints in EPG-Web unable to communicate with EPG-App?"

    The AI assistant keeps track of:

    • Tenant name
    • VRF
    • Bridge Domain
    • EPG names
    • Applied contracts
    • Recent APIC fault messages
    • Results of previous checks

    Instead of asking for the same information repeatedly, it builds on the conversation until the issue is resolved.


    Long-Term Memory – Your Cisco ACI Knowledge Base

    Long-term memory stores persistent information that does not disappear after the conversation ends. In AI systems, this often includes documentation and runbooks that can be retrieved when needed.

    For Cisco ACI, this could include:

    • Fabric architecture diagrams
    • Tenant standards
    • Naming conventions
    • Interface policies
    • L3Out design guides
    • APIC backup procedures
    • Security policies
    • Change management documents

    Cisco ACI Example

    An engineer asks:

    "What is our standard configuration for external routed networks?"

    Instead of relying on generic knowledge, the AI searches your organization's approved ACI design guide and provides recommendations based on your own standards.

    This improves consistency and reduces the risk of configuration drift.


    What is RAG?

    RAG (Retrieval-Augmented Generation) allows an AI assistant to retrieve relevant documentation before generating a response. Rather than guessing, it searches trusted sources such as your ACI runbooks, design guides, and operational procedures.

    Cisco ACI Example

    You ask:

    "How do we normally configure BGP authentication on our L3Outs?"

    The AI retrieves the organization's approved implementation guide and answers using that document rather than generic internet advice.


    Episodic Memory – Learning from Previous Incidents

    Episodic memory records previous interactions and user preferences so future assistance can be more personalized.

    Cisco ACI Example

    Suppose that last month you resolved a fault caused by a missing contract between two EPGs.

    Months later, a similar fault occurs.

    The AI recognizes the similarity and suggests checking contracts early in the troubleshooting process, saving valuable time.

    It may also remember that you prefer CLI outputs alongside APIC GUI navigation, allowing responses to match your working style.


    Bringing It All Together

    Imagine a production outage affecting application connectivity.

    An AI assistant could:

    1. Use Working Memory to keep track of the current troubleshooting session.
    2. Use Long-Term Memory to retrieve your organization's ACI standards and runbooks.
    3. Use Episodic Memory to recognize similar past incidents and apply successful troubleshooting patterns.

    This combination provides faster diagnostics, more consistent recommendations, and reduced troubleshooting time.


    Benefits for Cisco ACI Engineers

    By combining AI memory with Cisco ACI, organizations can:

    • Accelerate root cause analysis.
    • Reduce repetitive troubleshooting.
    • Retrieve design documentation instantly.
    • Improve adherence to operational standards.
    • Preserve knowledge from experienced engineers.
    • Shorten onboarding time for new team members.
    • Enable more intelligent AI-driven network operations.

    Key Takeaways

    • Working Memory manages the current troubleshooting context.
    • Long-Term Memory stores ACI documentation, standards, and runbooks.
    • Episodic Memory captures previous incidents and user preferences.
    • RAG connects AI to trusted enterprise documentation instead of relying only on model training.
    • Together, these capabilities can significantly improve Cisco ACI operations and troubleshooting efficiency.

    Frequently Asked Questions

    Can AI remember my Cisco ACI fabric permanently?
    Only if the AI platform is designed to store and retrieve persistent knowledge such as documentation and previous interactions.

    How does RAG help Cisco ACI administrators?
    It allows AI to search approved ACI documentation and generate answers based on your organization's standards instead of generic information.

    Can AI replace Cisco ACI engineers?
    No. AI augments engineers by reducing repetitive tasks and surfacing relevant information, while design decisions and operational oversight remain with experienced professionals.


    Related Articles to add to "How AI Memory Can Revolutionize Cisco ACI Operations":

    Need help with Cisco ACI, Nexus, data center networking, or network automation?

    I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.

    Contact me for consulting, troubleshooting, design reviews, and project support.


    Friday, 31 July 2026

    BGP Timers on a VRF in ACI Multi-Site — Why NDO Can't Do It and What Cisco TAC Confirmed

     If you've gone looking for a VRF-level BGP Timer Policy in Nexus Dashboard Orchestrator and come up empty, you're not missing a menu — it genuinely isn't there yet. Here's what we found while troubleshooting this on a live Multi-Site fabric with a mix of stretched and local VRFs, the workaround Cisco TAC confirmed as supported, and exactly how to verify the new timers actually took effect instead of just assuming they did.

    • The gap: NDO does not expose a BGP Timer Policy at the VRF level for ACI Multi-Site — only at the L3Out node level (via L3Out Node Routing Policy → BGP Node Settings).
    • The workaround: Create and apply the BGP Timer Policy directly on APIC, per site, against the target VRF (local or stretched). Cisco TAC confirmed this is the correct and supported method.
    • The catch: BGP timers are negotiated during the initial handshake. Changing the policy alone does nothing to an already-established session — you need a hard reset of the BGP session for the new keepalive/hold values to take effect.
    • Don't trust the config alone. "Configured" timers and "negotiated" (active) timers are two different things in show bgp output — always verify the negotiated value after the reset, not just the configuration.

    The Problem: No VRF-Level BGP Timer Option in NDO

    In a single-site APIC deployment, applying a custom BGP Timer Policy to a VRF is straightforward — it's a native object under Tenant → Networking → VRF → BGP Context Per Address Family. In Multi-Site environments managed through NDO, that granularity isn't there. NDO's L3Out configuration model lets you attach a BGP Node Routing Policy (with its own keepalive/hold timers) to a node group inside an L3Out template — but there's no equivalent construct for applying timers at the VRF level, independent of a specific L3Out's node groups.

    For environments running mixed local and stretched VRFs across sites, that's a real gap: you may want consistent BGP timer behavior tied to the VRF itself, not scattered across every L3Out node group that happens to reference it.

    What We Asked Cisco TAC

    Rather than guess, we opened a case and asked directly. Here's the exchange, included for anyone hitting the same wall:

    Our question:

    We need to deploy BGP timers on a VRF within our ACI Multi-Site fabric. Our environment contains both stretched and local VRFs. We do not see an option in NDO to apply BGP timers at the VRF level. Therefore, we are planning to use the following approach: 1) Create a BGP Timer Policy on the APIC. 2) Apply the BGP Timer Policy to the target VRF (local or stretched) via APIC only. Could you please confirm whether the above procedure is the correct and supported method for configuring BGP timers in an ACI Multi-Site environment? Additionally, after applying the policy, we do not see the updated timer values reflected on the BGP neighbour sessions. Is a BGP session reset (hard reset) required for the new timer values to take effect, or are there any additional steps that need to be performed?

    Cisco's response:

    Configuration Method: Your proposed approach is correct. As NDO does not currently support configuring BGP Timer Policies at the VRF level, applying these policies directly via the APIC on each respective site is the standard and supported method. 

    Session Reset: Yes, a BGP session reset (hard reset) is required for the new timer values to take effect. Because these timers are negotiated during the initial BGP handshake, the existing sessions must be torn down and re-established to renegotiate the new keepalive and hold-down values with the neighbor.

    Two things worth underlining from that response: this is a per-site, APIC-side configuration — NDO is not the tool for VRF-level BGP timers today — and a hard reset is mandatory, not optional, for the values to actually apply.

    Step-by-Step: Applying the BGP Timer Policy to a VRF via APIC

    1. Log into the APIC GUI for the site containing the target VRF (do this per site for stretched VRFs — there's no single-pane push for this from NDO).
    2. Navigate to Tenants → <tenant> → Networking → VRFs → <vrf-name>.
    3. Under the VRF, go to the BGP Context Per Address Family (create one for each address family in use — IPv4 and/or IPv6).
    4. Set the BGP Timers field to a custom BGP Timer Policy, or create a new one:
      • Go to Tenants → <tenant> → Policies → Protocol → BGP → BGP Timers, right-click and Create BGP Timers Policy.
      • Set Keepalive Interval (sec) and Hold Interval (sec) to your target values (keep the standard 1:3 ratio — e.g., 4/12 or 10/30 — most implementations expect hold time to be roughly 3x keepalive).
    5. Apply the policy to the VRF's BGP Context Per Address Family and submit.
    6. Repeat per site for any stretched VRF — this is not propagated automatically across the Multi-Site domain since it's an APIC-local object, not an NDO-managed one.
    7. Confirm the object landed correctly:
       GET https://<apic>/api/node/mo/uni/tn-<tenant>/ctx-<vrf>/bgpCtxAfP-ipv4-ucast.json

    Check for the tnBgpTimersPolName attribute pointing to your new policy.

    Why the Timers Won't Show Up Until You Reset the Session

    This trips people up every time: BGP keepalive and hold timers are exchanged in the OPEN message during session establishment — they're not a live, continuously-enforced parameter that updates on the fly. Changing the policy updates the configuration, but an already-established session keeps running on whatever timers it originally negotiated until it's torn down and rebuilt.

    That means after step 7 above, show bgp will show your configured timers, but the negotiated/active timers on any existing session will still be the old values — until you force a reset.

    Verifying the Change Actually Took Effect

    Don't just trust the APIC MO or the CLI's "configured" line — verify the negotiated value, which is what's actually governing the session.

    1. Check the configured value on the border leaf:

    show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

    Look for: Configured hold time is 12, keepalive interval is 4 seconds

    2. Check the negotiated (active) value in the same output: Look for: Hold time is X, keepalive interval is Y seconds If X/Y still show the old defaults, the session hasn't renegotiated — move to step 3.

    3. Force the hard reset:

    clear bgp <afi> unicast <peer-ip> vrf <tenant>:<vrf-name>

    A soft clear will not renegotiate timers — timers only get exchanged at session establishment, so this has to be a hard clear.

    4. Re-check the negotiated value after the reset:

    show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name>

    The negotiated value will be the lower of what each side offers — if your peer's own BGP timer configuration is set higher (or lower) than yours, that governs the final negotiated result, not your configured value alone.

    5. Confirm real-world keepalive cadence:

    show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last read"

    Run this two or three times a few seconds apart — "Last read" should reset roughly every N seconds, matching your new keepalive interval.

    6. Watch for flapping after the change, especially with aggressive timers like 4/12:

    show bgp <afi> unicast neighbors <peer-ip> vrf <tenant>:<vrf-name> | include "Last reset"
    show logging logfile | include BGP

    A Word of Caution on Aggressive Timers

    Tighter timers (like 4-second keepalive / 12-second hold) improve failure-detection speed, but they raise the risk of false-positive flaps on links with variable latency, high CPU load on the peer device, or congested control-plane paths. Before rolling this out broadly across a Multi-Site fabric, test it on a single non-critical VRF/L3Out first and monitor for a few days.

    FAQ

    Does NDO support VRF-level BGP timers at all? Not currently. The only BGP timer construct exposed in NDO is at the L3Out node group level (L3Out Node Routing Policy → BGP Node Settings). VRF-level BGP Timer Policies must be applied directly on APIC, per Cisco TAC's confirmation.

    Will this survive an NDO template push or resync? Since the VRF-level BGP Timers Policy is applied directly on APIC and not modeled in the NDO template, it should not be overwritten by a standard template redeploy — but any workflow that deletes and recreates the VRF from NDO would remove it. Treat it as an out-of-band configuration and document it clearly for your team.

    Do I need to reset the session on both sides of the peering, or just my side? Just your side's session needs a hard clear to force renegotiation — BGP will tear down and rebuild the full session, which inherently re-triggers the handshake and timer exchange on both ends.

    What's a safe keepalive/hold ratio to use? Cisco and most vendors expect hold time to be roughly 3x keepalive (e.g., 4/12, 10/30, 20/60). Going below that ratio, or setting a very low hold time on a link with real latency or jitter, increases false-positive flap risk.

    Is this the same procedure for a stretched VRF across multiple sites? Yes, but it must be repeated per site — this is an APIC-local object, and NDO doesn't propagate it across the Multi-Site domain automatically. For a stretched VRF, log into each site's APIC and apply the same BGP Timer Policy independently.

    Related Reading on Networklearner


    Author note: 

    Need help with Cisco ACI, Nexus, data center networking, or network automation?
    I am a CCIE Data Center engineer with 18+ years of enterprise networking experience.
    Contact me for consulting, troubleshooting, design reviews, and project support.