If you've ever pulled up interface counters and seen one port-channel member carrying 70–80% of the traffic while its bundle-mate sits nearly idle, your first instinct is probably the same one we had: "BGP load balancing must be broken."
It usually isn't. Here's the case we worked through, why BGP was innocent from the start, and the actual fix.
TL;DR
- Symptom: One port-channel member interface was carrying the majority of traffic; the other was underutilized.
- First suspect: BGP
maximum-paths— already configured correctly, and confirmed working via ECMP. - Real cause: The port-channel load-balancing algorithm (
src-dst-ipenhanced) was hashing large, persistent flows onto the same physical link every time. - Fix: Move to
src-dst-mixed-ip-port(or the NX-OS equivalentsrc-dst ip-l4port) to add Layer 4 entropy to the hash. - Caveat: If the imbalance is caused by a single "elephant flow," even L4 hashing won't split it — that needs a different tool entirely.
The Symptom
The environment was already running router bgp with maximum-paths 8 configured, so multiple equal-cost paths were expected to be installed and used. But interface counters told a different story — one member of the port-channel was consistently running hot while the other stayed quiet, day after day, regardless of overall traffic volume.
Router# show interface port-channel 10 | include rate
5 minute input rate 812000000 bits/sec, 95000 packets/sec
5 minute output rate 790000000 bits/sec, 92000 packets/sec
Router# show interface gigabitethernet 1/1 | include rate
5 minute input rate 780000000 bits/sec, 91000 packets/sec
Router# show interface gigabitethernet 1/2 | include rate
5 minute input rate 32000000 bits/sec, 4000 packets/secThat's roughly a 96/4 split across two members of the same bundle — not something you'd expect from a healthy port-channel.
Ruling Out BGP First
Before touching anything on the switching side, we confirmed the routing layer was actually doing its job.
Router# show ip bgp summary
Router# show ip route 10.20.30.0
Routing entry for 10.20.30.0/24
Known via "bgp 65001", distance 20, metric 0
Routing Descriptor Blocks:
* 192.168.1.1, from 192.168.1.1
192.168.1.2, from 192.168.1.2Two next-hops installed for the same prefix, exactly as maximum-paths 8 should produce when the paths are equal-cost. BGP ECMP was functioning correctly — routing decisions were being made across multiple paths, so it made no sense to keep chasing BGP config as the root cause.
That's the key thing to internalize: BGP ECMP and port-channel load balancing are two completely separate mechanisms operating at two different layers.
| Layer | Mechanism | Decides |
|---|---|---|
| Layer 3 (Routing) | BGP maximum-paths / ECMP | Which path (next-hop) traffic takes |
| Layer 2 (Switching) | Port-channel hash algorithm | Which physical member link within a bundle carries a given flow |
Even with BGP perfectly distributing traffic across paths, each of those paths might egress through a port-channel — and it's entirely possible for the hashing algorithm on that port-channel to funnel most flows onto one member link regardless of what BGP is doing upstream.
Checking the Current Hashing Algorithm
Next step: find out what hash algorithm the port-channel was actually using.
On IOS / IOS-XE:
Router# show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ipOn Nexus (NX-OS):
Switch# show port-channel load-balance
Port Channel Load-Balancing Configuration:
System: src-dst ipBoth confirmed the same thing: the algorithm was using only Layer 3 information — source and destination IP address — to compute the hash and select a member link.
Why src-dst-ip Pins Big Flows to One Link
The hash is deterministic. Given the same source IP and destination IP, the algorithm produces the same result every single time, which means every packet in that flow is sent down the same physical link.
That's fine when you have many different flows between many different IP pairs — the hash spreads them out reasonably well. It falls apart when a small number of flows dominate the traffic, which is exactly what a database replication stream, a backup job, or a storage sync between two fixed IPs looks like: same source, same destination, forever — one hash result, one link, permanently.
This isn't a bug. Cisco's port-channel load balancing is intentionally flow-based, not packet-based, specifically to avoid packet reordering within a TCP session. The trade-off is that a big, long-lived flow can dominate a single member link even when the bundle as a whole has spare capacity.
The Fix: Add Layer 4 to the Hash
Switching from src-dst-ip to src-dst-mixed-ip-port pulls TCP/UDP source and destination port numbers into the hash calculation alongside the IP addresses.
IOS / IOS-XE:
Router(config)# port-channel load-balance src-dst-mixed-ip-portNX-OS (Nexus):
Switch(config)# port-channel load-balance src-dst ip-l4portSome Nexus platforms go a step further and support VLAN as an additional hash input for even more entropy:
Switch(config)# port-channel load-balance src-dst ip-l4port-vlanWith source/destination ports added, two different application sessions between the same pair of servers (say, a web server and a database server running several concurrent connections) will now very likely produce different hash results — and land on different physical links — instead of all piling onto whichever link the IP-only hash happened to pick.
In environments with many concurrent sessions between the same endpoints (application servers, load balancers, hypervisor clusters), this alone often meaningfully improves the balance.
Before You Make the Change
A few things worth knowing before you touch a production bundle:
- It's generally non-disruptive on most Cisco platforms — you don't need to shut the port-channel.
- Every existing flow gets re-hashed immediately. Expect a brief window of packet reordering as flows redistribute across members. It's usually transparent to TCP, but schedule the change for a maintenance window or low-traffic period if the environment is sensitive (VoIP, storage replication, etc.).
- This is still flow-based hashing, just with more inputs. A single elephant flow — one enormous, sustained stream between one IP:port pair — will still be pinned to one physical link even after adding L4 information. Layer 4 hashing splits up multiple flows between the same hosts; it does not split a single flow.
If It's an Elephant Flow, Hashing Alone Won't Fix It
If, after the change, one link is still running noticeably hotter than the others, don't keep tweaking the hash algorithm — look for a dominant flow instead:
Router# show interface port-channel 10 | include rate
Switch# show flow monitor <name> cacheor pull top talkers from NetFlow/sFlow/telemetry. If you find one flow consuming most of the bandwidth, your real options are traffic engineering (splitting the application across multiple sessions/IPs if possible), platform-specific dynamic load balancing features (several Nexus platforms support Dynamic Load Balancing, which rebalances flows based on real-time link utilization rather than a static hash), or simply accepting that a single-flow workload isn't a great fit for port-channel-based scaling in the first place.
Verification Checklist
After changing the algorithm, give it a few hours under real traffic and check:
- Port-channel member interface utilization (
show interface … | include rateon each member) - Top bandwidth-consuming flows (NetFlow/sFlow/telemetry)
- Interface error/discard counters (rule out anything unrelated masquerading as imbalance)
- Application-level traffic distribution if you have flow visibility tools
If the split moves from something like 96/4 toward 55/45 or 60/40, the change did its job. If it barely moves, you're very likely looking at an elephant-flow scenario rather than a hashing-algorithm problem.
FAQ
Does changing the port-channel load-balancing algorithm cause an outage? No — on most Cisco IOS, IOS-XE, and NX-OS platforms this is a non-disruptive change. Existing flows get re-hashed and may briefly reorder, but the port-channel itself stays up.
Will src-dst-mixed-ip-port always produce perfectly even traffic distribution?
No. It improves distribution when multiple flows exist between the same endpoints, but a single very large flow will still be pinned to one member link, since hashing is flow-based, not packet-based.
How do I check the current load-balancing algorithm on a Nexus switch?
show port-channel load-balance on NX-OS. On IOS/IOS-XE, use show etherchannel load-balance.
Is BGP maximum-paths related to port-channel hashing at all?
No — they operate at different layers. maximum-paths controls how many equal-cost paths BGP installs at Layer 3. Port-channel hashing controls which physical member link, at Layer 2, carries a given flow once it's already been routed onto that path.
What's the best load-balancing algorithm for Cisco port-channels in general?
There isn't a universal "best" — it depends on traffic patterns. src-dst-mixed-ip-port (or NX-OS's ip-l4port) is a strong general-purpose default for environments with many concurrent sessions between the same hosts, since it adds the most useful entropy without added complexity.
Conclusion
BGP maximum-paths was never the issue in this case — it was correctly installing and using multiple equal-cost paths exactly as designed. The imbalance lived one layer down, in the port-channel's src-dst-ip hashing algorithm, which pins any given source/destination IP pair to a single physical link. Moving to src-dst-mixed-ip-port added Layer 4 awareness to the hash and meaningfully improved distribution — with the caveat that a genuine elephant flow needs a different fix entirely.
The broader lesson: when you see uneven link utilization on a bundle, don't assume the routing protocol is at fault. Check where the imbalance actually lives — Layer 3 path selection or Layer 2 hash distribution — before you start changing BGP configuration that may already be working exactly as intended.
Related Articles
If this was useful, these go deeper on the BGP and Cisco fundamentals that sit underneath this issue:
