CCIE DC: Two advanced vPC Concepts.

Hi Guys!


Today we are going to chat about some advanced vPC concepts, I have done vPC to death but I want to chat about something that bothers me with vPC and peoples understanding I think of vPC.


When you enable a VLAN for vPC, (which you do by sending it across the vPC Peer Link) your making quite a commitment here. vPC only really works when you think about all the aspects of what your implementing.


OK, so let's say you have a fairly simple topology, something like this:






Here we have the kind of topology you should never really implement, which is a vPC where there are hosts who are not dual-attached, this is very bad as we will soon see why.


Here is a basic idea of the configuration:


N5K-p1-2# show run | sect vpc
feature vpc
vpc domain 1
  peer-keepalive destination 10.0.8.201 source 10.0.8.202
  vpc peer-link
N5K-p1-2#

N5K-p1-2# show run int po1

!Command: show running-config interface port-channel1
!Time: Sun Jul  7 06:09:17 2013

version 5.1(3)N1(1)

interface port-channel1
  switchport mode trunk
  switchport trunk allowed vlan 10
  spanning-tree port type network
  speed 10000
  vpc peer-link!

N5K-p1-2# show ip int brief
IP Interface Status for VRF "default"(1)
Interface            IP Address      Interface Status
Vlan1                169.254.1.12    protocol-up/link-up/admin-up
Vlan10               169.254.10.2    protocol-up/link-up/admin-up
N5K-p1-2# show run | inc route
  ip route 0.0.0.0/0 10.0.8.254
ip route 0.0.0.0/0 169.254.1.1
N5K-p1-2#
This config is pretty much the same obviously with the IP addressing differing for N5k1.

The config I am trying to convey is: N5k1 and N5k2 are the access layer, they have a vPC enabled VLAN, VLAN 10, that is going to face towards our end hosts, and upstream towards the 7k is the rest of the network, which is the 7k's, which are all in VLAN 1.


On the N7k:

N7K-1-1(config-if-range)# show ip int brief
IP Interface Status for VRF "default"(1)
Interface            IP Address      Interface Status
Vlan1                169.254.1.1     protocol-up/link-up/admin-up
Lo1                  3.3.3.3         protocol-up/link-up/admin-up
3.3.3.3 is going to be our internet, we are going to use this to help us work out if there are problems.


So as you can see it's fairly straight forward at the moment, but we are now going to do some things to show why it's so incredibly important that you DUAL ATTACH your equipment when your dealing with vPC.

First let's configure our vPC down to the host


interface port-channel10
  switchport access vlan 10
  spanning-tree port type edge
  speed 1000
  vpc 10Pretty simple so far right? This is all the vPC config we are used to. Nothing special so far, Great!

Let's look at a few things, let's examine a show vpc on the primary:


N5K-p1-1(config-if)#     show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 1
Peer status                     : peer adjacency formed ok
vPC keep-alive status           : peer is alive
Configuration consistency status: success
Per-vlan consistency status     : success
Type-2 consistency status       : success
vPC role                        : primary
Number of vPCs configured       : 1
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     10

vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        up     success     success                    10

Here we can see that the N5k1 is the primary vPC and that the member vPC po10 is all up and happy, let's try disabling an interface on the host who is configured for vPC and see what happens, do we still have connectivity?

We have disabled port 6, which goes to N5k2, which is the secondary, what happens:






We still have ping, that is good, what else can we tell?

5K-p1-2(config-if)# show vpc 10
vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        down*  success     success                    -
N5K-p1-2(config-if)#

So the port has been shown as down on N5k-2 because we have physically disabled that port on the host itself, so far so good, let's look at a few other things.

N5K-p1-1(config-if)# show vpc orphan-ports
Note:
--------::Going through port database. Please be patient.::--------
Note this port is not listed as an orphan port, we will find out later what _exactly_ an orphan port is and how important it is.

Ok, let's try something now, what if we disable the peer link between N5k1 and N5k2, we would expect traffic to continue flowing.

N5K-p1-1(config-if)# int po1
N5K-p1-1(config-if)# shut
N5K-p1-1(config-if)# show vpc

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    down   -

vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        up     success     success                    10

Traffic continues to flow, pretty much exactly what we would hope for and expect so far, so our dual connected server is surviving pretty well so far!

Let's keep looking, let's re-enable the port and try disabling the port to the PRIMARY vPC:






Another succesfull ping, again everything is going super well so far. Let's try disabling the vPC peer-link again:

N5K-p1-2(config-if)# int po1
N5K-p1-2(config-if)# shut
2013 Jul  7 06:29:56 N5K-p1-2 %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary

Oh dear...





Looks like the peer link being down is enough to force the secondary to turn off it's own links, why? to avoid a dual-active scenario!

Let's see what happens if we actually reload the switch instead


5K-p1-1# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y

We then see this error message on N5k2:

----------------------------------------------------------------------------
--More--2013 Jul  7 06:33:08 N5K-p1-2 %VPC-2-PEER_KEEP_ALIVE_RECV_FAIL: In domain 1, VPC peer keep-alive receive has failed

But, more importantly, our port remains up:


We still don't see it as an orphan port:

N5K-p1-2# show vpc orphan-ports
ERROR: Peer-link is down


This to me helps cement something, in this situation, where we have a host who is dual-connected to two switches, a vPC orphan-port is NOT a port where his peer link has failed, or a port where the other vPC Member port on the other switch has failed, no no no, this is not what an orphan-port is, an orphan-port is something entirely diffirent which we will get to now.


OK, Let's say that Host3 in our diagram is being connected to the switch, here is his port config on N5k2 (because he is only attached to N5k2):


N5K-p1-2(config)# int eth1/12
N5K-p1-2(config-if)# switchport access vlan 10
N5K-p1-2(config-if)# no shut
N5K-p1-2(config-if)# spanning-tree port type edge trunk
Pretty simple config hey? Now this, _THIS_ is an orphan port.


N5K-p1-2(config-if)# show vpc orphan-ports
Note:
--------::Going through port database. Please be patient.::--------

VLAN           Orphan Ports
-------        -------------------------
10             Eth1/12

There's your proof guys, _this_ is an orphan port, Why? Because its a port, that is not configured as a vPC member port, that is in a VLAN enabled for vPC!

This shows on the output for this command too:

N5K-p1-2(config-if)# show vpc ?
  orphan-ports            Show ports that are not part of vPC but have common
                          VLANs
So you see, THIS is what an orphan port actually is.


An orphan port exhibits some particular behaviors, for example, we saw before that if we shut down the peer-link, that ports would turn off that are only attached to the N5k2 as a dual-active protection mechanism, let's see what happens this time to this orphan port.

N5K-p1-2(config)# int po1
N5K-p1-2(config-if)# shut
2013 Jul  7 06:57:44 N5K-p1-2 %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary

N5K-p1-2# show int eth1/12
Ethernet1/12 is up

The port remains up, because it is an Orphan Port! Yahoo!

OK guys, so far so good right? Well there is as always with Cisco, a little more to it.

So in our example, Eth1/12 is up, which means host 3 is up and happy.. except Host 3 has a major problem: he is totally isolated from the network, because vlan 10 is down:

N5K-p1-2(config-if)# show int vlan 10
Vlan10 is down, line protocol is down
  Hardware is EtherSVI, address is  0005.73ba.637c
  Internet Address is 169.254.10.2/24
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec

VLAN 10 is down, because it is a vPC enabled VLAN SVI Interface, which are brang down along with all the vPC member interfaces.

So our friend Host3 is now totally isolated! He can't ping the internet!



Note that this is only because we shut down the peer-link but not the keepalive, so this VLAN was brang down because it's part of the dual-active protection mechanism: if we had just reloaded switch1, the VLAN would have stayed up.

We could tell VLAN 10 not to suspend as part of a dual-active scenario, let's do so:



N5K-p1-2(config-if)# vpc domain 1
N5K-p1-2(config-vpc-domain)# dual-active ?
  exclude  Interface vlans to exclude from suspension when dual-active

N5K-p1-2(config-vpc-domain)# dual-active exclude ?
  interface-vlan  Interface vlan configuration

N5K-p1-2(config-vpc-domain)# dual-active exclude interface-vlan 10
Warning:
 !! Will overwrite previous interface-vlan exclude config!!

Once the vPC adjacency is back up, we will see this message in the logs:

N5K-p1-2(config-vpc-domain)# 2013 Jul  7 07:02:22 N5K-p1-2 %VPC-2-VPC_SVI_EXCLUDE: vPC 10 has up vlans 10 which are excluded from suspension on dual-active


Pretty good so far, let's see what happens now if we shut down the link:


N5K-p1-2(config-if)# int po1
N5K-p1-2(config-if)# shut

2013 Jul  7 07:04:23 N5K-p1-2 %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going down, suspending all vPCs on secondary
 
So far so good, let's see if our interfaces are up:

N5K-p1-2(config-if)# show int eth1/12
Ethernet1/12 is up

N5K-p1-2(config-if)# show int vlan 10
Vlan10 is up, line protocol is up
  Hardware is EtherSVI, address is  0005.73ba.637c


Looks good! can our host get to the net?



Wooooo Hoo!

We are golden, but we have a few things that could have gone horribly wrong.


Let's pretend for a minute that instead of a normal every day host being attached to each of the 5k's, Replace host 1 and host 3 with some sort of network device that has redundant interfaces, but does not support port-channels (ASA firewall for example, not sure if the new version of ASA software supports etherchannels, maybe it does, If so please leave a comment, but for our purposes just pretend it's a device that doesn't do port channels)

For this to make sense, we will also need to change our topology a little, so it looks a bit more like this:


(I am no artist! If there was a CCIE of drawing and Diagrams I would definately not pass hahaha, my friend Paul Tursan would be a Quadruple CCIE in that though)

But the idea is, N5k2's only method to get to the rest of the network is via his Peer-link (crappy setup, but again, to make this example work we need to have a config like this)



Ok so here is my device that doesn't do port channels, here is the config on each switch:



interface Ethernet1/12
  switchport access vlan 10
  spanning-tree port type edge trunk
  speed 1000This is now an orphan-port on BOTH of our switches:

 N5K-p1-2(config-if)# show vpc orphan-ports
Note:
--------::Going through port database. Please be patient.::--------

VLAN           Orphan Ports
-------        -------------------------
10             Eth1/12

N5K-p1-1(config-if)# show vpc orphan-ports
Note:
--------::Going through port database. Please be patient.::--------

VLAN           Orphan Ports
-------        -------------------------
10             Eth1/12

Now, let's pretend our firewall thinks that the link going to N5k2 is it's active link, the link going to N5k1 is a standby link.


Our Peer-link Dies! What happens?


N5K-p1-2(config-if)# int po1
N5K-p1-2(config-if)# shut


N5K-p1-2(config-if)# show int eth1/12
Ethernet1/12 is up
The interface stays up! In our topology this would be very bad, as Now the firewall would be totally isolated: Firewall would still think that the interface up to N5k2 is active and working, but as he forwards traffic up to it, it would be black-holed, instead what we want is for if the peer link dies, the secondary vPC to suspend the port:



N5K-p1-2(config-if)# vpc orphan-port ?
  suspend  Suspend - when vPC secondary peerlink goes down
I hope this helps someone out there better understand vPC!