Hi Guys.
There is a lot of confusion in my own mind for FCoE Multihop, I remember back in the day that the argument was "well, you can do FCoE with the Nexus 2k's which is not true multihop but it kind of is!" I also remember that this blog post: http://brasstacksblog.typepad.com/brass-tacks/2011/06/fcfcoe-connectivity-options-as-of-6272011.html by Erik was one of the best posts on the topic. I have asked Erik Via Twitter to update this diagram as a lot of it has changed now in regards to Cisco UCS.
In this blog post I will show one of the examples given by Erik, which is multihop FCOE Configuration on the Nexus 5k's in the hope that it will help someone out there.
Authors note: I think that the order of operation is very important when configuring multihop FCoE as I had major issues getting this going the first time I attempted it. My advice is to get the absolute basic FCoE Connectivity going first, then add complexity once your comfortable that it's all configured correctly.
Steps:
After enabling FCoE Ofcourse with feature FCOE on your nexus 5k,The first thing to do on both your Nexus 5k's is configure the appropriate VLAN's and VSAN's.
vsan database
vsan 10
!
vlan 10
fcoe vsan 10
vlan 20
name DataVLAN
In this example I have configured VSAN 10, and bound it to VLAN 10.
Next, I need to configure the ethernet interface between my two switches appropriately:
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10
!
At this point I personally still don't "no shut" the interfaces and I wait until I have configured everything.
The next step is to configure the VFC Interface:
interface vfc1
bind interface Ethernet1/10
switchport mode E
switchport trunk allowed vsan 10
no shutdown
!
Now this is done, I no shut the ethernet interface on each end:
int eth1/10
no shut
!
Now let's look at our VFC1 interface:
switch# show int vfc1
vfc1 is trunking
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 216 bits/sec, 27 bytes/sec, 0 frames/sec
1 minute output rate 200 bits/sec, 25 bytes/sec, 0 frames/sec
3667 frames input, 440724 bytes
0 discards, 0 errors
3790 frames output, 521116 bytes
0 discards, 0 errors
last clearing of "show interface" counters Sun Apr 21 04:54:33 2013
Interface last changed at Sun Apr 21 05:34:21 2013
Success! Our VSAN is trunking across the link, a show fcns database helps us verify this:
switch# show fcns database
VSAN 10:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0f08d1 NL 21:00:00:0a:60:55:17:69 (Seagate) scsi-fcp:target
[disk]
0x0f08d2 NL 21:00:00:11:d6:3e:ee:2e scsi-fcp:target
[disk1]
Finally an FCPING helps us verify 100 percent.
switch# fcping pwwn 22:00:00:04:cf:21:a5:2e vsan 10
28 bytes from 22:00:00:04:cf:21:a5:2e time = 1759 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 287 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 222 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 304 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 270 usec
At this point we have an FCoE Multihop topology.
If all you came for is how to configure an FCoE Trunk between the two 5k's you can stop reading now as now we are getting into a bit more detail
So the first thing I asked myself is, does this VLAN run spanning tree?
ToSanSWITCH# show spanning-tree vlan 10
Spanning tree instance(s) for vlan does not exist.
Alright, that answers that question, but what happens if i assign a normal ethernet port to this FCoE VLAN?:
ToSanSWITCH(config)# int eth1/1
ToSanSWITCH(config-if)# switchport access vlan 10
ToSanSWITCH(config-if)# end
What does my show spanning-tree say now?
ToSanSWITCH# show spanning-tree vlan 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32778
Address 547f.eeaf.3a3c
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 547f.eeaf.3a3c
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/1 Desg FWD 2 128.129 P2p
So according to this, the VLAN is forwarding on Eth1/1, at first I was quite confused and wondered to myself what kind of effect this would have on the FCoE Traffic, but it occured to me that the ethertype for FCoE Traffic is diffirent, what I imagine would happen is that Eth1/1 would receive copies of the FCoE Frames, but if the device plugged in at Eth1/1 has no idea what those frames are (i.e. doesn't recognise or want to use the ethertype) chances are it will just ignore them.
Regardless, this configuration had no ill effect on my VFC Interface as it was still showing as up:
ToSanSWITCH# show int vfc1
vfc1 is trunking
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:3a:3f
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
I decided to see if I could convince it to "break", the first thing I tried was adding a VSAN to this interface that was not configured with an appropriate FCoE VLAN, which in this case was VSAN 1
I configured this on both ends:
switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1
The worst that happens is that the VSAN 1 stays in isolation mode:
switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1
Next, I configured on one of the switches a port facing a server:
interface Ethernet1/1
switchport mode trunk
switchport trunk native vlan 20
spanning-tree port type edge trunk
!
interface vfc10
bind interface Ethernet1/1
switchport trunk allowed vsan 10
no shutdown
!
vsan database
vsan 10 interface vfc10
!
The server port comes up fine:
switch# show int vfc10
vfc10 is trunking
Bound interface is Ethernet1/1
Hardware is Ethernet
Port WWN is 20:09:54:7f:ee:af:1c:bf
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 10
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
At this point I am determined to make it my mission to break this FCoE Link as when I first tried to set this up, I had major problems, I want to see what configuration or what order of configuration you have to do for it NOT to work.
Some of you may have noticed that despite what the cisco configuration guide mentions I did NOT have to add these trunk interfaces to my vsan database, i.e. i did NOT need the following configuration:
vsan database
vsan 10 interface vfc1
I decide to add this config to see if this kills the VFC, it does not:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
So at this point I have not been able to get the VFC not to work for VSAN 10, it does not work for VSAN 1 but that is because I do not have an equivilant FCoE VLAN for that VSAN, which I can totally understand.
Next I try making the Ethernet interface carry both LAN and SAN Traffic:
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10,20
switch# show spanning-tree int eth1/10
Vlan Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0020 Desg FWD 2 128.138 P2p
This does not faze the device and the VFC interface remains up:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
At this point I decide to up the ante by adding another interface:
interface Ethernet1/20
switchport mode trunk
switchport trunk allowed vlan 10,20
!
interface vfc2
bind interface Ethernet1/20
switchport mode E
no shutdown
!
This new interface comes up just fine:
switch# show int vfc2
vfc2 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/20
Hardware is Ethernet
Port WWN is 20:01:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) (1)
I even get nice load balancing across the links:
switch# show fspf database vsan 10
FSPF Link State Database for VSAN 10 Domain 0xd8(216)
LSR Type = 1
Advertising domain ID = 0xd8(216)
LSR Age = 341
LSR Incarnation number = 0x80000013
LSR Checksum = 0x79e0
Number of links = 3
NbrDomainId IfIndex NbrIfIndex Link Type Cost
-----------------------------------------------------------------------------
0x0f(15) 0x00040000 0x00040000 1 125
0x9a(154) 0x001e0000 0x001e0000 1 100
0x9a(154) 0x001e0001 0x001e0001 1 100
So at this point, everything is working pretty well. I decide to try making the link between the two switches a port-channel:
feature lacp
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10,20
channel-group 20 mode active
!
interface Ethernet1/20
switchport mode trunk
switchport trunk allowed vlan 10,20
channel-group 20 mode active
!
interface vfc1
bind interface port-channel20
switchport mode E
switchport trunk allowed vsan 10
no shutdown
!
The VFC Int is still up and carrying traffic:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is port-channel10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
At this point I don't understand why I had trouble getting the FCoE VE Int to come up the first time but am willing to accept that the feature works perfectly and it must have been some sort of order of operation issue.
Thanks!!
ReplyDeleteGreat post Peter :)
ReplyDeletejust a quick one I really need to know is the following:-
1. If there is a already a principal Switch in the domain, how to set/configure another switch as subordinate switch?
2. Is it required to configure or both switch will negotiate them self using their FCID/FCdomain ?
3. If they failed and became isolated for each vlan how to configure them as subordinated switch??
4. Is there any order of operation to set them up or any configuration required?
Please let me know thanks
Hey there,
DeleteTo answer some of your questions:
1. Make sure that the new FC switch in the fabric is configured with an fcdomain priority lower than the current Principal switch in the fabric. Cisco MDS and N5Ks come with a default priority of 128. So if your fabric is Cisco-based, just configure "fcdomain priority 1 vsan 20" for each VSAN you want your FC switch to be a part of.
2. I'm not sure what you are referring to, but for vfc interfaces that will be used for MultiHop FCoE, you need to specify "switchport mode e". For vfc interfaces that go to hosts you don't have to change the mode (default is F port). Each FC switch in the fabric will be assigned an fcdomain ID by the Principal switch, but you can request a "prefered" ID like this "fcdomain domain 0x51 prefered vsan 20". If the principal has that ID within the allowed range and it's not taken by another switch, it will grant it, otherwise you'll get the next available ID.
3. For the new switch that comes in the fabric to not be isolated, make sure that you don't configure a STATIC fcdomain ID that is already taken. And also make sure that both E ports are configured with the same VSAN. Other than that, I'd say you would only need to worry about the zoning merge.
4. Like Peter said, it's always good to leave the physical ports in shutdown state and only bring them up after you've finished all the required configuration. It will save time.
Hope that helped.
Of course you must know it by now, but this may help others:
ReplyDeleteThe reason you don't need to add that multihop FCoE vfc (VE port) to the vsan database is that it is by default set to vsan 1. And, unless you suspend VSAN 1, there's no need to place your E / VE ports in any other VSAN. E / VE ports just need that "linked" VSAN to be able to come up, but then they will happily trunk all allowed VSANs.