CCIE DC: First Official Rack Rental! Spanning-tree Bridge Assurance and LACP suspend-individual

Hi Guys!

Today marked an important occasion as I had my first "official" rack rental (I have had others thanks to some generous people, but this was my first with the "big two" training vendors)

I concentrated on:

- CFS
- Port Channels (LACP)
- Spanning-tree Bridge Assurance
- FEX stuff


Here is the first few useful things i found.


I am sure we have all done this before:


You configure a few member interfaces for your etherchannel:


SW3(config)# int eth1/4, eth1/2
SW3(config-if-range)# channel-group 2 mode active


You configure a few options for the Port channel:

interface port-channel2
  description ### I am L33t ###
  switchport mode trunk
  switchport trunk allowed vlan 1
  spanning-tree port type network
  speed 10000


These then apply on your member ports:

interface Ethernet1/4
  switchport mode trunk
  switchport trunk allowed vlan 1
  channel-group 2 mode active


But woops you forgot a port, you meant to add Eth1/1 too!



SW3(config)# int eth1/1
SW3(config-if)# channel-group 2 mode active
command failed: port not compatible [Ethernet Layer]


Damn what a pain in the ass! Now I have to go and add all the options to the port, like the spanning-tree mode etc.. or do I?

 SW3(config-if)# channel-group 2 force mode active


Let's take a look at the config now:


 SW3(config-if)# show run int eth1/1
 
interface Ethernet1/1
  switchport mode trunk
  switchport trunk allowed vlan 1
  channel-group 2 mode active


Awesome! All the appropriate config has applied without me having to put it all in manually. A bit of a time saver on the Lab, when every second will count!

Let's talk more about LACP, there are two commands available for LACP that are not on non-nexus platforms, and these commands are enabled by default, and they can actually be quite a pain:

lacp suspend-individual

and

lacp graceful-convergence


Let's talk about suspend-individual.

so the idea behind LACP suspend-individual is that if a port-channel does not receive any LACP PDU's on a particular port-channel, in the normal case these ports would be placed into "Individual" state:


SW1# show port-channel sum
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SD)     Eth      LACP      Eth3/1(I)    Eth3/3(I)   

 

This means that LACP will treat these as two independent links, not as an etherchannel, but let's say you had the other end of the link misconfigured, and had port channel in ON mode on the other end:


SW2# show run int eth1/1

interface Ethernet1/1
  switchport mode trunk
  channel-group 10


Suddenly you have a potential loop in the network, and you will see some very strange spanning-tree behavior:

SW2# show spanning-tree vlan 30

VLAN0030
  Spanning tree enabled protocol rstp
  Root ID    Priority    4126
             Address     547f.eec2.7d01
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    4126   (priority 4096 sys-id-ext 30)
             Address     547f.eec2.7d01
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10             Desg FWD 1         128.4105 P2p
SW2# show spanning-tree vlan 30

VLAN0030
  Spanning tree enabled protocol rstp
  Root ID    Priority    4126
             Address     547f.eec2.7d01
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    4126   (priority 4096 sys-id-ext 30)
             Address     547f.eec2.7d01
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po10             Desg BLK 1         128.4105 Dispute P2p



The port will block/unblock as it keeps seeing "Designated" BPDU's.

We can resolve this by telling the port on the upstream switch to suspend ports if they are part of an etherchannel and we are expecitng to receive LACP PDU's for them:


SW1(config)# int po1
SW1(config-if)# lacp suspend-individual
ERROR: Cannot set/reset lacp suspend-individual for port-channel1 that is admin up
SW1(config-if)# shut
SW1(config-if)# lacp suspend-individual
SW1(config-if)# no shut



Now the ports will show as suspended:

SW1(config-if)# end
SW1# show port-channel sum
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SD)     Eth      LACP      Eth3/1(s)    Eth3/3(s)   


As soon as we reconfigure our etherchannel correctly on the other side:


SW2(config)# int eth1/1, eth1/3
SW2(config-if-range)# channel-group 10 mode active

The port comes out of suspended state and traffic will flow

SW1# show port-channel sum
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth3/1(P)    Eth3/3(P)   

So as you can see, you didn't need to no shut the ports or anything on the Sw2 side, you just had to get it to start advertising LACP PDU's

The PROBLEM is that some linux hosts and some other hosts, will not bring up the LACP until they receive the LACP PDU's first, so this can make the switch place the ports into the suspended state indefinately, since the switch is expecting LACP PDU's, but the host never sends them, so the port channel remains down.

So for linux hosts and other devices that may not send PDU's straight away, turn off LACP suspend individual, for all other ports your perfectly safe having it enabled.


Let's talk about bridge assurance.


Bridge assurance is a feature on the Nexus platforms that uses BPDU's as a method to perform "pruning" of unwanted VLAN's (although this is more of an unforseen benefit of the design) and to protect against unidirectional links.

The way bridge assurance works is, if you specify a port as spanning-tree port type network (which is NOT set by default by the way, except on vPC Peer-Links) then what will happen is spanning-tree bridge assurance will force both links to constantly send BPDU's both directions, as sort of a method of keepalive, if spanning-tree bridge assurance notices that these BPDU's go missing on either end, it knows that there is a unidirectional fault on the link (or another fault on the link) and immediately blocks the port via spanning-tree so that an alternative path can be taken.

The added advantage of this technology, is that when using rapid spanning-tree, each VLAN has it's own BPDU's right? Let's say we have a config like this:



Switch 1 has VLAN 1, 10, and 30

Switch 3 has VLAN 1 and 10


On the switches port-channels to each other, we specify these are "network" ports:



SW3# show run int po2

interface port-channel2
  description ### I am L33t ###
  switchport mode trunk
  switchport trunk allowed vlan 1
  spanning-tree port type network
  speed 10000
!

SW1# show run int po2
interface port-channel2
  switchport
  switchport mode trunk
  spanning-tree port type network

As you can see here, we have port type network on both switches, lets see what spanning-tree on Sw1 has to say about this:

SW1# show spanning-tree int po2

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Desg FWD 1         128.4097 Network P2p

VLAN0010         Desg BKN*1         128.4097 Network P2p *BA_Inc

VLAN0030         Desg BKN*1         128.4097 Network P2p *BA_Inc


As you can see, spanning-tree bridge assurance is blocking vlan 10 and 30 from going out on this link because on sw3 we have said switchport trunk allowed vlan 1, so the upstream swithc (Swithc 1) is not receiving any BPDU's, so as  far as he is concerned there's no good reason to send the traffic down.

If we add VLAN 10 to the Switch 3 trunk interface:



SW3(config)# int po2
SW3(config-if)# switchport trunk allowed vlan add 10


This will instantly as soon as the BPDU's are advertised unblock the port on the upstream switch:


SW1# 2013 May 12 10:02:47 SW1 %$ VDC-1 %$ %STP-2-BRIDGE_ASSURANCE_UNBLOCK: Bridge Assurance unblocking port port-channel2 VLAN0010.

Now we have seen how spanning-tree bridge assurance works, let's see what can happen if we misconfigure it.


In this example, we have a trunk between SW1 to SW2:

SW1# show run int po1


interface port-channel1
  switchport
  switchport mode trunk
  spanning-tree port type network

On Switch2:

interface port-channel10
  switchport mode trunk
  speed 10000

Just to make this example a bit easier to follow, on Switch 2 we have made Switch 2 the root of the spanning tree.

Let's take a look at the show spanning-tree on Switch 1:

SW1# show spanning-tree interf po1

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Desg BKN*1         128.4096 Network P2p *BA_Inc

VLAN0010         Desg BKN*1         128.4096 Network P2p *BA_Inc

VLAN0030         Root FWD 1         128.4096 Network P2p 


As you can see from the above example, the switch has placed the port into BLOCKING based on Bridge Assurance (BA_INC).

It has however kept vlan 30 unblocked, why?


SW2# show spanning-tree inter po10

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0001         Root FWD 1         128.4105 P2p

VLAN0030         Desg FWD 1         128.4105 P2p



Because for VLAN 30, SW2 is the root switch. Since SW2 is the root for VLAN 30, that means that the port facing upstream is a designated port from SW2's perspective, BPDU's are always sent out Designated ports, therefore SW1 is receiving BPDU's from SW2:


SW2# debug spanning-tree bpdu_tx

2013 May 12 06:59:11.492578 stp: RSTP(30): transmitting RSTP BPDU on port-channel10
2013 May 12 06:59:11.492608 stp: vb_vlan_shim_send_bpdu(1977): VDC 1 Vlan 30 port port-channel10 enc_type 1 len 42
2013 May 12 06:59:13.492584 stp: RSTP(30): transmitting RSTP BPDU on port-channel10
2013 May 12 06:59:13.492615 stp: vb_vlan_shim_send_bpdu(1977): VDC 1 Vlan 30 port port-channel10 enc_type 1 len 42
2013 May 12 06:59:15.492581 stp: RSTP(30): transmitting RSTP BPDU on port-channel10
2013 May 12 06:59:15.492610 stp: vb_vlan_shim_send_bpdu(1977): VDC 1 Vlan 30 port port-channel10 enc_type 1 len 42
2013 May 12 06:59:17.490094 stp: RSTP(30): transmitting RSTP BPDU on port-channel10
2013 May 12 06:59:17.490124 stp: vb_vlan_shim_send_bpdu(1977): VDC 1 Vlan 30 port port-channel10 enc_type 1 len 42
2013 May 12 06:59:19.490097 stp: RSTP(30): transmitting RSTP BPDU on port-channel10
2013 May 12 06:59:19.490127 stp: vb_vlan_shim_send_bpdu(1977): VDC 1 Vlan 30 port port-channel10 enc_type 1 len 42



As you can see from the above debug output, SW2 is sending BPDU's out po10 on vlan 30, since SW1 is receiving BPDU's for this VLAN, the bridge assurance feature says well im receiving BPDU's, so we are good to go here, lets unblock this port.


What is missing from this output though is SW2 sending BPDU's for VLAN 1 and 10, it will NOT send these, why? because for VLAN 1 and 10 Port10 is SW2's root port (the port where it can find the root bridge) and spanning-tree does not transmit BPDU's up the root port, therefore spanning-tree bridge assurance is not receiving any BPDU's for these VLAN's and is therefore blocking the port.


This shows the importance that if you are going to use spanning-tree bridge assurance, you need to make sure you set the spanning-tree port type network on BOTH ends of the link, if your connecting to a 6500 for example, you can't do spanning-tree bridge assurance, therefore you want to turn it off for any ports facing a 6500. (or just don't specify spanning-tree port type network, because it will only run on ports configured as spanning-tree port type network)


Let's fix up the spanning-tree port type network on switch 2:

SW2(config)# int po10
SW2(config-if)# spanning-tree port type network

As soon as we do this, SW1 unblocks:


SW1# 2013 May 12 10:14:05 SW1 %$ VDC-1 %$ %STP-2-BRIDGE_ASSURANCE_UNBLOCK: Bridge Assurance unblocking port port-channel1 VLAN0001.