CCIE DC: The VPC Topology you never want to implement, the "Pink Slip" Topology

Hi Guys

So one of my most popular blog posts of all time is my "vPC - The gotcha's you need to know", with a whole.. 10.. maybe even 15!!! unique visitors, no I kid it's a few more than that and easily the most popular on the site.


Let's take a fresh approach to my blog here and write about a topology for vPC that you would NEVER want to implement, it will be funny for us to identify the mistakes made in this topology, plus we can learn all about vPC Failure scenario's too!

Just a few quick shout out's, A certain someone who if they want to take credit will post a reply is responsible for getting me access to a lab so i can actually show you all this, So cheers Anonymous Person!

The INE CCIE DC Video's where invaluable in the creation of this blog post, the advanced vPC video's are exceptionally good.

The blog post will contain "ALWAYS" tips that you can use to make sure your implementing a secure, robust vPC Topology first time every time.

It's worth noting that almost every step of the way, protection mechanisms in NXOS and vPC prevented me from trying to configure it the wrong way, for example, I was unable to configure the peer link before I configured the keepalive.

Anyway now that's out the way let's look at our Topology.









(As you can clearly see, I was actually meant to be a graphic artist but got lumped into networking ;))

OK so now we have our simple diagram showing our very simple topology. Our Hero Network Engineer, let's call him leethax0r decides to implement a vPC Topology, "how hard can it be?" He says to himself, After all I have seen the Cisco Certified guys do it tons of times, it's just like port-channel's with a stacked switch right? Of course leethax0r is too proud to learn how to do properly, if it was up to him, they'd be implementing an ABC (Anything But Cisco) Network anyway.

Through a series of serious mistakes our hero leethax0r (or hax0r for short) implements a terrible configuration, leading to a terrible vPC implementation (Which invaribly, he will blame on "cisco bugs", rather than his own misunderstanding and misconfiguration)


The first thing hax0r decides to do, is implement a peer keepalive, he knows this is the first step in implementing vPC


(Config shown in red so that hopefully people don't copy paste it and try and implement it!)

feature interface-vlan
feature lacp
feature vpc
int vlan 10
 no shut
 ip add 10.10.10.2/24
!





hax0r does the same thing on each side and changes the IP Address, so far he hasn't really done anything wrong, It's perfectly valid to use an SVI interface for your peer keepalives, although it is strongly recommended to place them in there own VRF so they are not affected by the global network IP routing, it's also recommended to create a dedicated link between the two switches that just carries this particular VLAN for this SVI, but this is where our hero hax0r goes horribly wrong...

Hax0r decides that he wants to implement OSPF and rely on the network upstream from his 5k to provide alternative paths for the peer keepalives to reach each other, what could possibly go wrong? he says to himself:

ALWAYS: Implement a totally seperate vRF for the vPC Peer keepalive and (where possible) have a back to back interface between the vPC Peers, a directly connected interface only carrying the peer keepalive, the peer-keepalive link prevents dual-active, the most catastrophic situation that can occur with vPC!

This is the config that hax0r implements:

Switch 2

interface Vlan10
  no shutdown
  ip address 10.10.10.2/24
  ip router ospf 1 area 0.0.0.0

interface Vlan20
  no shutdown
  ip address 10.20.20.1/24
  ip router ospf 1 area 0.0.0.0
 

Switch 1:
interface Vlan10
  no shutdown
  ip address 10.10.10.1/24
  ip router ospf 1 area 0.0.0.0

interface Vlan20

interface Vlan30
  no shutdown
  ip address 10.30.30.1/24
  ip router ospf 1 area 0.0.0.0


He then checks he can reach switch 1 from Switch 2 on there seperate VLAN interfaces that will be routed via VLAN 10 (the shared VLAN)

SIWTCH2# ping 10.30.30.1
PING 10.30.30.1 (10.30.30.1): 56 data bytes
64 bytes from 10.30.30.1: icmp_seq=0 ttl=254 time=0.823 ms
64 bytes from 10.30.30.1: icmp_seq=1 ttl=254 time=0.619 ms
64 bytes from 10.30.30.1: icmp_seq=2 ttl=254 time=0.61 ms
64 bytes from 10.30.30.1: icmp_seq=3 ttl=254 time=7.755 ms
64 bytes from 10.30.30.1: icmp_seq=4 ttl=254 time=9.645 ms
L33thax0r pats himself on the back at his correct implementation of OSPF.

Next, hax0r implements the vPC Peer Keepalive:

vpc domain 1
  peer-keepalive destination 10.20.20.1 source 10.30.30.1 vrf default
!

hax0r checks show vpc to see if the peer is up


SIWTCH2# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer link not configured
vPC keep-alive status             : peer is aliveConfiguration consistency status  : failed
Per-vlan consistency status       : failed
Configuration inconsistency reason: vPC peer-link does not exist
Type-2 consistency status         : failed
Type-2 inconsistency reason       : vPC peer-link does not exist
vPC role                          : none established
Number of vPCs configured         : 0
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Disabled (due to peer configuration)
Auto-recovery status              : Disabled

Hax0r is now happy that his vPC is showing as up, hax0r, out of his depth notices something about a "peer link", some quick googling leads hax0r to some sample configuration, so hax0r implements the config he found on a popular ABC podcast


interface Ethernet1/20
  switchport mode trunk
  channel-group 1 mode active
!
SIWTCH2(config)# int po1
SIWTCH2(config-if)# vpc peer-link

This is dead wrong: Hax0r didn't use more than one link for the peer-link and  he didn't set the spanning-tree port type to network (Thankfully this last one the switch will do this automatically for him)

ALWAYS: Implement multiple links, bundled together for your vPC Peer-link, if your using a Nexus 7000 this should be across multiple linecards (and if you don't have multiple linecards in your nexus 7000, GET THEM!)

ALWAYS: Use spanning-tree port type network on the vPC Peer Link (Switch will do this automatically for you)

Hax0r now checks the vPC Output:

SIWTCH2# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : primary
Number of vPCs configured         : 0
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    up     1,10

Hax0r has done it! The VPC is up! Truly his l33tness should  be known everywhere, maybe he'll even be able to convince the bosses to take out this evil Cisco network and put ABC vendor in, now he's shown his l33t skills.

However, little known to hax0r, his shoddy vPC config is a timebomb waiting to go off.

L33t Hax0r calls his co-worker and system administrator, , LulzAdmin (or Lulz for short) to come and plug his servers in, the vPC is ready.

LulzAdmin (Who is almost as good as l33t hax0r at his job) has his server ready. The server is MISSION Critical, the entire business depends on this server.

l33t hax0r and lulzadmin can't work out how to dual attach there server, they can't seem to get the teaming working across two ports, and after blaming "cisco bugs" they decide that the server will be fine single attached.

ALWAYS: Dual Attach your servers to both vPC Peers, dual attach EVERYTHING, you never want single-attached devices.
 
This mission critical server, is about to have some major problems...

L33t hax0r plugs the server into one of the ports on the switch:

int eth1/1
 channel-group 10 mode active

!
int po10
 switchport mode access
 switchport access vlan 50
!


Examine the below output:


SWITCH1# show vpc orphan-ports
Note:
--------::Going through port database. Please be patient.::--------

VLAN           Orphan Ports
-------        -------------------------
1              Eth1/10, Eth1/15, Eth2/1, Eth2/2
10             Eth1/10
50             Eth1/1, Eth1/10

Here is our first problem, as mentioned, we have some single attached ports, which in this case is vPC 50,



SIWTCH2# show mac address-table vlan 10
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 10       547f.eeaf.1cbc    static    0          F    F  Router
* 10       547f.eeaf.3a3c    static    0          F    F  Po1
 



What is wrong with this output? Let's see, the OSPF neighbor relationship between the two switches is via the Port-Channel link! That means if the vPC Peer link goes down, the keepalives will also go down as well, which means the vPC Secondary will suspend any vPC Peer Ports for risk of a dual active situation!


After configuring this mess, hax0r and lulz go for a "Hard earnt" drink at the local bar, meanwhile, the Data Centre they are hosting the Nexus at has a scheduled Power Outage for UPS B Feed, if you have your equipment dual attached to two power supplies on two diffirent feeds, you should be good, but take a guess as to if our hero hax0r did such a thing ;).

The Secondary nexus, connected to UPS Feed B, powers off.

Lulz and Hax0r receive a phone call, it's BigBossMan, the Webserver has gone Offline! What have you done?


Lulz and Hax0r rush to the data centre to investigate the problem, upon logging into the powered up nexus, Lulz and Hax0r see the following:


SWITCH1# show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1
Peer status                       : peer link is down
vPC keep-alive status             : Suspended (Destination IP not reachable)Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : secondary, operational primary
Number of vPCs configured         : 1
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    down   -

vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        down   failed      Peer-link is down          -SWITCH1#
The port channel to the server! Po10! It's down, Why? What is this "peer link down" all about, just because the other switch died, why should this one fail?

The problem is that the vPC peer link can't come up until the peer keepalive is up, and the peer keepalive can't come up until the vPC Peer Link comes up... Even though VLAN 10 is configured to go upstream to the rest of the network (as per hax0r's plan to use the rest of the global routing table to allow the keepalives to function) when a VPC Peer-link goes down, any SVI's that belong to that vPC Peer-link will shut down.

ALWAYS: Make sure the L3 link which the two vPC peers use for there routing protocol adjacency  does NOT travel over the vPC Peer-link, as if the vPC peer link dies, these SVI's will be suspended!




Lulz and hax0r, do some quick googling and seem to think they need a command called "auto-recovery, as quickly as they can they configure auto-recovery on the switches and reload the switches...

NOW, they truly are hosed, by implementing auto-recovery which tells both switches to go active after a reload if they can't see the peer, and by not having the peer-link able to come up because the keepalives can't get through, they will now cause a dual-active scenario.

The Dual Active Scenario leads the entire network to come down, Cisco Certified Engineers come in and inspect what happened, proving the misconfiguration on hax0r's part, hax0r is summarily shown his pink slip, still cursing "cisco bugs" for his downfall.


This is just some of the mistakes that can be made when your configuring vPC that can lead it to cause major problems, Now that we know what we shouldn't do, let's examine the diffirent commands in VPC and what they protect us from.

First of all, I explain the peer gateway command in detail in my vPC - The gotcha's you need to know article. You should always implement peer-gateway, especially if you have netapp or F5 Devices.

What about Auto-Recovery, what exactly does that Do?

As we explained in our previous example, auto-recovery is useful in a situation where both switches power off, but only one switch turns back on (maybe the other one was hit by an electrical surge or something else), in this particular scenario, if auto-recovery was not turned on what would happen is that the vPC would never establish, so the switch that is now ON would never turn on it's vPC member port,s because the vPC peer-link would never have come up.

Auto-recovery will resolve this, Auto-recovery says that after a certain period (default is 240 seconds), the switch will assume the peer has died and bring up the ports.

The Most important thing, when enabling Auto-recovery, is to be damn sure that if both switches reset, they will always be able to get the vPC Peer Link up and/or the vPC Peer Keepalive so that they can detect a dual-active scenario.

Therefore, it is recommended to turn on auto-recovery, and as long as you can satisfy the above criteria, you are safe to turn this on.

NOTE: You can turn on auto-recovery retroactively, so if you ever walked into a situation where this was occuring and you hadn't turned  it on previously, if you turn it on, 240 seconds later the vPC will become active.

Auto Recovery also assists in another situation, if you have a vPC Peer-link go down:

SWITCH2(config)# int po1
SWITCH2(config-if)# shut
2013 May 18 15:23:35 SWITCH2 %$ VDC-1 %$ %VPC-2-VPC_SUSP_ALL_VPC: Peer-link going
down, suspending all vPCs on secondary

The Secondary will suspend all vPC Member Ports as it should, this is the behavior that is executed as part of vPC to prevent Dual-Active Scenario's and to ensure correct forwarding, because the peer-link is quite important.

So, now what happens if the primary now dies in this situation, the secondary vPC will STILL leave the ports down:


vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po1    down   -

vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        down   failed      Peer-link is down          -

With auto-recovery, the secondary switch can realise that the primary is not coming back and enable the vPC member ports.

Again however, it's exceptionally important that your positive the vPC Peer Keepalive will always work reliably so as to avoid dual active scenario's, this is the reason that auto-recovery is not enabled by default, because it could potentially lead to dual-active.


The following output is typically what you will see when a vPC PO has been enabled due to auto-recovery:

vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
10     Po10        up     success     Type checks were bypassed  50
                                      for the vPC


Let's look at two other commands, graceful consistency check and Peer-switch.

Graceful consistency check helps in the following situations, Let's pretend our friend hax0r has been rehired at a company, much to his chagrin they have vPC, our friend hax0r is asked to reconfigure a vPC port-channel from an access port to a trunk port.

Woops! the ports have just gone down now as the paramters don't match, our friend hax0r is not having much luck!

With the graceful-consistency check command configured, one end of the link will suspend but the other will remain up, giving you chance to match the parameters on both side, so you don't have to bring the link down to make changes, therefore this is highly recommended.


With Peer-Switch, unfortunately in my topology I do not have enough switches to show you what particular scenario is needed for it to be helpful, but to cut a long story short, if you have a switch connected via a vPC port channel to your two vPC Peers, if one of these Peers is the root of the spanning-tree, Peer-switch can be useful to ensure that in the event of failure of this root bridge, and subsequent recovery, that there is no delay in forwarding while waiting for spanning-tree to reconverge, because both the vPC Peers will appear as one giant bridge from the perspective of spanning-tree (since they will share the same bridge-ID), I recommend you turn this command on, i cannot think of any reason not to.


So In Conclusion, I personally recommend the following as default config for vPC:

Graceful Consistency Check
Auto Recovery (Just be sure your peer-keepalive or peer-link will always work after a reboot)
Peer-Gateway
Peer-Switch
 

Finally, there are a few interesting options for your vPC that could allow you to do some naughty configuration if you wanted to.


The Dual-active command under vPC Config allows you to specify that a particular SVI interface will NOT go down if the peer-link fails, but instead is excluded from suspension, this could have been useful to our friend hax0r for VLAN 10, but it's still not recommended to configure it this way, you should not need to use this dual-active command in most situations, except in certain orphan port situations where you want to ensure that if an orphan port exists, and the peer-link dies, that the orphan port on the secondary can get to it's SVI interface.


I hope you enjoyed this blog post I spent a while on it, I hope I covered off your questions :)