CCIE DC: Multihop FCoE on Nexus 5k

Hi Guys.

There is a lot of confusion in my own mind for FCoE Multihop, I remember back in the day that the argument was "well, you can do FCoE with the Nexus 2k's which is not true multihop but it kind of is!" I also remember that this blog post: http://brasstacksblog.typepad.com/brass-tacks/2011/06/fcfcoe-connectivity-options-as-of-6272011.html by Erik was one of the best posts on the topic. I have asked Erik Via Twitter to update this diagram as a lot of it has changed now in regards to Cisco UCS.

In this blog post I will show one of the examples given by Erik, which is multihop FCOE Configuration on the Nexus 5k's in the hope that it will help someone out there.

Authors note: I think that the order of operation is very important when configuring multihop FCoE as I had major issues getting this going the first time I attempted it. My advice is to get the absolute basic FCoE Connectivity going first, then add complexity once your comfortable that it's all configured correctly.

Steps:
After enabling FCoE Ofcourse with feature FCOE on your nexus 5k,The first thing to do on both your Nexus 5k's is configure the appropriate VLAN's and VSAN's.

vsan database
 vsan 10
!
 
vlan 10
  fcoe vsan 10

vlan 20
  name DataVLAN


In this example I have configured VSAN 10, and bound it to VLAN 10.

Next, I need to configure the ethernet interface between my two switches appropriately:

interface Ethernet1/10
  switchport mode trunk
  switchport trunk allowed vlan 10

!

At this point I personally still don't "no shut" the interfaces and I wait until I have configured everything.

The next step is to configure the VFC Interface:


interface vfc1
  bind interface Ethernet1/10
  switchport mode E
  switchport trunk allowed vsan 10
  no shutdown

!

Now this is done, I no shut the ethernet interface on each end:

int eth1/10
no shut
!
 
Now let's look at our VFC1 interface:


switch# show int vfc1
vfc1 is trunking
    Bound interface is Ethernet1/10
    Hardware is Ethernet
    Port WWN is 20:00:54:7f:ee:af:1c:bf
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Trunk vsans (admin allowed and active) (10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()
    1 minute input rate 216 bits/sec, 27 bytes/sec, 0 frames/sec
    1 minute output rate 200 bits/sec, 25 bytes/sec, 0 frames/sec
      3667 frames input, 440724 bytes
        0 discards, 0 errors
      3790 frames output, 521116 bytes
        0 discards, 0 errors
    last clearing of "show interface" counters Sun Apr 21 04:54:33 2013

    Interface last changed at Sun Apr 21 05:34:21 2013



Success!  Our VSAN is trunking across the link, a show fcns database helps us verify this:


switch# show fcns database

VSAN 10:
--------------------------------------------------------------------------
FCID        TYPE  PWWN                    (VENDOR)        FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0f08d1    NL    21:00:00:0a:60:55:17:69 (Seagate)       scsi-fcp:target
                  [disk]
0x0f08d2    NL    21:00:00:11:d6:3e:ee:2e                 scsi-fcp:target
                  [disk1]




Finally an FCPING helps us verify 100 percent.


switch# fcping pwwn  22:00:00:04:cf:21:a5:2e vsan 10
28 bytes from 22:00:00:04:cf:21:a5:2e time = 1759 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 287 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 222 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 304 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 270 usec


At this point we have an FCoE Multihop topology.


If all you came for is how to configure an FCoE Trunk between the two 5k's you can stop reading now as now we are getting into a bit more detail


So the first thing I asked myself is, does this VLAN run spanning tree?


ToSanSWITCH# show spanning-tree vlan 10
Spanning tree instance(s) for vlan does not exist.


 Alright, that answers that question, but what happens if i assign a normal ethernet port to this FCoE VLAN?:

ToSanSWITCH(config)# int eth1/1
ToSanSWITCH(config-if)# switchport access vlan 10
ToSanSWITCH(config-if)# end


What does my show spanning-tree say now?

ToSanSWITCH# show spanning-tree vlan 10

VLAN0010
  Spanning tree enabled protocol rstp
  Root ID    Priority    32778
             Address     547f.eeaf.3a3c
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32778  (priority 32768 sys-id-ext 10)
             Address     547f.eeaf.3a3c
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/1           Desg FWD 2         128.129  P2p



So according to this, the VLAN is forwarding on Eth1/1, at first I was quite confused and wondered to myself what kind of effect this would have on the FCoE Traffic, but it occured to me that the ethertype for FCoE Traffic is diffirent, what I imagine would happen is that Eth1/1 would receive copies of the FCoE Frames, but if the device plugged in at Eth1/1 has no idea what those frames are (i.e. doesn't recognise or want to use the ethertype) chances are it will just ignore them.


Regardless, this configuration had no ill effect on my VFC Interface as it was still showing as up:

ToSanSWITCH# show int vfc1
vfc1 is trunking
    Bound interface is Ethernet1/10
    Hardware is Ethernet
    Port WWN is 20:00:54:7f:ee:af:3a:3f
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Trunk vsans (admin allowed and active) (10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()



I decided to see if I could convince it to "break", the first thing I tried was adding a VSAN to this interface that was not configured with an appropriate FCoE VLAN, which in this case was VSAN 1

I configured this on both ends:

 
switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1


The worst that happens is that the VSAN 1 stays in isolation mode:


switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1


 Next, I configured on one of the switches a port facing a server:

interface Ethernet1/1
  switchport mode trunk

  switchport trunk native vlan 20
  spanning-tree port type edge trunk


!

interface vfc10
  bind interface Ethernet1/1
  switchport trunk allowed vsan 10
  no shutdown

!

vsan database
  vsan 10 interface vfc10

!


The server port comes up fine:

switch# show int vfc10
vfc10 is trunking
    Bound interface is Ethernet1/1
    Hardware is Ethernet
    Port WWN is 20:09:54:7f:ee:af:1c:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 10
    Trunk vsans (admin allowed and active) (10)
    Trunk vsans (up)                       (10)



At this point I am determined to make it my mission to break this FCoE Link as when I first tried to set this up, I had major problems, I want to see what configuration or what order of configuration you have to do for it NOT to work.

Some of you may have noticed that despite what the cisco configuration guide mentions I did NOT have to add these trunk interfaces to my vsan database, i.e. i did NOT need the following configuration:

vsan database
  vsan 10 interface vfc1




I decide to add this config to see if this kills the VFC, it does not:



switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
    Bound interface is Ethernet1/10
    Hardware is Ethernet
    Port WWN is 20:00:54:7f:ee:af:1c:bf
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 10
    Trunk vsans (admin allowed and active) (1,10)
    Trunk vsans (up)                       (10)



So at this point I have not been able to get the VFC not to work for VSAN 10, it does not work for VSAN 1 but that is because I do not have an equivilant FCoE VLAN for that VSAN, which I can totally understand.

 Next I try making the Ethernet interface carry both LAN and SAN Traffic:


interface Ethernet1/10
  switchport mode trunk
  switchport trunk allowed vlan 10,20

switch# show spanning-tree int eth1/10

Vlan             Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0020         Desg FWD 2         128.138  P2p


This does not faze the device and the VFC interface remains up:


switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
    Bound interface is Ethernet1/10
    Hardware is Ethernet
    Port WWN is 20:00:54:7f:ee:af:1c:bf
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 10
    Trunk vsans (admin allowed and active) (1,10)
    Trunk vsans (up)                       (10)


At this point I decide to up the ante by adding another interface:


interface Ethernet1/20
  switchport mode trunk
  switchport trunk allowed vlan 10,20

!
 interface vfc2
  bind interface Ethernet1/20
  switchport mode E
  no shutdown

!


This new interface comes up just fine:

switch# show int vfc2
vfc2 is trunking (Not all VSANs UP on the trunk)
    Bound interface is Ethernet1/20
    Hardware is Ethernet
    Port WWN is 20:01:54:7f:ee:af:1c:bf
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Trunk vsans (admin allowed and active) (1,10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             (1)



I even get nice load balancing across the links:

switch# show fspf database vsan 10

FSPF Link State Database for VSAN 10 Domain 0xd8(216)
LSR Type                = 1
Advertising domain ID   = 0xd8(216)
LSR Age                 = 341
LSR Incarnation number  = 0x80000013
LSR Checksum            = 0x79e0
Number of links         = 3
 NbrDomainId      IfIndex   NbrIfIndex    Link Type         Cost
-----------------------------------------------------------------------------
    0x0f(15) 0x00040000     0x00040000               1          125
   0x9a(154) 0x001e0000     0x001e0000               1          100
   0x9a(154) 0x001e0001     0x001e0001               1          100

So at this point, everything is working pretty well. I decide to try making the link between the two switches a port-channel:





feature lacp

 
interface Ethernet1/10
  switchport mode trunk
  switchport trunk allowed vlan 10,20
  channel-group 20 mode active

 !
interface Ethernet1/20
  switchport mode trunk
  switchport trunk allowed vlan 10,20
  channel-group 20 mode active

!
interface vfc1
  bind interface port-channel20
  switchport mode E

  switchport trunk allowed vsan  10
  no shutdown

!

The VFC Int is still up and carrying traffic:


switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
    Bound interface is port-channel10
    Hardware is Ethernet
    Port WWN is 20:00:54:7f:ee:af:1c:bf
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 10
    Trunk vsans (admin allowed and active) (1,10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()


At this point I don't understand why I had trouble getting the FCoE VE Int to come up the first time but am willing to accept that the feature works perfectly and it must have been some sort of order of operation issue.










4 comments:

  1. Great post Peter :)
    just a quick one I really need to know is the following:-
    1. If there is a already a principal Switch in the domain, how to set/configure another switch as subordinate switch?
    2. Is it required to configure or both switch will negotiate them self using their FCID/FCdomain ?
    3. If they failed and became isolated for each vlan how to configure them as subordinated switch??
    4. Is there any order of operation to set them up or any configuration required?
    Please let me know thanks

    ReplyDelete
    Replies
    1. Hey there,

      To answer some of your questions:

      1. Make sure that the new FC switch in the fabric is configured with an fcdomain priority lower than the current Principal switch in the fabric. Cisco MDS and N5Ks come with a default priority of 128. So if your fabric is Cisco-based, just configure "fcdomain priority 1 vsan 20" for each VSAN you want your FC switch to be a part of.

      2. I'm not sure what you are referring to, but for vfc interfaces that will be used for MultiHop FCoE, you need to specify "switchport mode e". For vfc interfaces that go to hosts you don't have to change the mode (default is F port). Each FC switch in the fabric will be assigned an fcdomain ID by the Principal switch, but you can request a "prefered" ID like this "fcdomain domain 0x51 prefered vsan 20". If the principal has that ID within the allowed range and it's not taken by another switch, it will grant it, otherwise you'll get the next available ID.

      3. For the new switch that comes in the fabric to not be isolated, make sure that you don't configure a STATIC fcdomain ID that is already taken. And also make sure that both E ports are configured with the same VSAN. Other than that, I'd say you would only need to worry about the zoning merge.

      4. Like Peter said, it's always good to leave the physical ports in shutdown state and only bring them up after you've finished all the required configuration. It will save time.

      Hope that helped.

      Delete
  2. Of course you must know it by now, but this may help others:

    The reason you don't need to add that multihop FCoE vfc (VE port) to the vsan database is that it is by default set to vsan 1. And, unless you suspend VSAN 1, there's no need to place your E / VE ports in any other VSAN. E / VE ports just need that "linked" VSAN to be able to come up, but then they will happily trunk all allowed VSANs.

    ReplyDelete

Popular old posts.