CCIE DC: Advanced FCOE Part 2

Hi Guys

For an even better guide on FCoE than I have done, check out this link:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/fcoe/513_n1_1/ops_fcoe.pdf

And this one
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html


 
Hot on the tails of Advanced FCoE Comes Advanced FCoE Part 2: Return of the Frame Format! Or something, I don't know comedy is not my strong point :p



Anyway, let's get to it.

So first thing i wanted to talk about, when we left our hero last we where chatting about FCoE and a few topologies, I am going to cover off a few I couldn't do previous

First of all, let's look at a very simple FCoE Config with vPC, it's a topic that makes people panic very quickly.

OK:
here is an example config that works perfectly:

N5k1:
feature vpc
vpc domain 1
  peer-keepalive destination 10.2.8.14
  vpc peer-link
  vpc 2

vlan 5
  name DATAVLAN
vlan 10
  fcoe vsan 10
  name SANA

interface Ethernet1/1
  switchport mode trunk
  switchport trunk allowed vlan 5,10
  spanning-tree port type edge trunk
  channel-group 2
!

interface port-channel2
  switchport mode trunk
  switchport trunk allowed vlan 5,10
  spanning-tree port type edge trunk
  speed 10000
  vpc 2
!


n5k1-1(config)# show run int vfc2

!Command: show running-config interface vfc2
!Time: Mon Jun 10 04:47:55 2013

version 5.2(1)N1(4)

interface vfc2
  bind interface Ethernet1/1  switchport trunk allowed vsan 10
  no shutdown 
!


N5k2:
 

interface Ethernet1/1
  switchport mode trunk
  switchport trunk allowed vlan 5,20
  spanning-tree port type edge trunk
  channel-group 2

 interface port-channel2
  switchport mode trunk
  switchport trunk allowed vlan 5,20
  spanning-tree port type edge trunk
  speed 10000
  vpc 2
 

interface vfc2
  bind interface port-channel2
  switchport trunk allowed vsan 20
  no shutdown
The VPC is still up and functioning:

5k1-1(config)# show vpc
Legend:
vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
2      Po2         up     success     success                    5,10      


In the above examples I highlighted that the vFC on N5k1 was bound to the member int, but on N5k2 it was bound to the port-channel, both options will work, but i recommend you bind to the PHYSICAL int, why?

THE MAIN REASON YOU WOULD DO THIS? There is a good reason

if you bind to the PHYSICAL INT, the port-channel doesn't have to come up first for the vPC to come up, which is CRUCIALLY important.

IF YOU BIND TO THE PORT CHANNEL, THEN LACP HAS TO COME UP BEFORE THE VFC COMES UP.. SO HOW WILL BOOT FROM SAN WORK?

Sorry for the caps, but When i saw that.. I was like wow

Ok let's move on to our final consideration, which is eVPC, this is very very easy, everyone panics about it, but the config is exactly the same as you would do above, obviously with the member interfaces being numbered diffirently.

The only thing you have to know, is that each FEX in your topology must have the FCoE Command, only one fex for each topology can have the FCoE Command on it, and it should be the one that is carrying the traffic for that SAN, this picture should help explain further:




(Source: http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/fcoe/513_n1_1/ops_fcoe.pdf)




This would be exactly the same with Adapter-FEX, as shown below:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/guide_c07-690080.html#wp9000411

"The relevant configuration for Cisco Nexus 5500-1 would look like this:

fex 105

pinning max-links 1

description "FEX0105"

type N2232P

vlan 11

fcoe vsan 11

vsan database

vsan 11

vsan 11 interface vfc11

interface vfc11

bind interface Vethernet11

no shutdown

vsan database

vsan 11 interface vfc11

interface Vethernet11

switchport mode trunk

switchport trunk allowed vlan 1,11

bind interface Ethernet105/1/2 channel 4

The relevant configuration for Cisco Nexus 5500-2 would look like this:

fex 106

pinning max-links 1

description "FEX0106"

type N2232P

vlan 12

fcoe vsan 12

vsan database

vsan 12

vsan 12 interface vfc12

interface vfc12

bind interface Vethernet12

no shutdown

vsan database

vsan 12 interface vfc12

interface Vethernet12

switchport mode trunk

switchport trunk allowed vlan 1,12

bind interface Ethernet106/1/2 channel 3

"






FIP VLAN Discovery

FIP VLAN discovery is a technology utilized by FCoE to allow an ENode to discover the FCoE VLAN from the FCF by specifically requesting it on a special multicast/broadcast address.

It's important to note so your not driven mental:

 "FIP VLAN discovery is an optional protocol in FC-BB-5. An ENode implementation can choose to offer only manual configuration for FCoE VLANs, and therefore choose not to perform FIP VLAN discovery. It is commonly assumed that such implementation will default to VLAN 1002 for its FCoE VLAN. The Cisco Nexus 5000 Series supports FIP VLAN discovery, and it will respond to any ENode that performs a query. The contents of the response depend on how the virtual FC interface is configured on the Cisco Nexus 5000 Series Switch, as discussed later in this document."

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html

it's OPTIONAL! Your adapter MAY NOT support it, some adapters do (like the qlogic, which i have verified with 100 percent certainty does support it:


http://filedownloads.qlogic.com/files/Manual/77172/Install_Guide_Unified_Fabric_Pilot.pdf

Also, the Operating system must support it, as witnessed here by VMWARE celebrating the fact they now have support for it in ESX 5.0
http://www.vmware.com/support/vsphere5/doc/vsp_esxi50_u2_rel_notes.html


Now, if you meet all those requirements, its important to note, you do NOT need to allow the native VLAN if you don't want to, (although to be honest, might not hurt), i have tested this multiple times, FIP VLAN Discovery traffic appears to, like CDP, be allowed to go down an interface even if the link has not been enabled for that particular VLAN, observe the following:


n5k1-1(config-if)# show run int eth1/1

!Command: show running-config interface Ethernet1/1
!Time: Mon Jun 10 03:42:38 2013

version 5.2(1)N1(4)

interface Ethernet1/1
  switchport mode trunk
  switchport trunk allowed vlan 5,30
  spanning-tree port type edge trunk


This port is attached to an adapter that supports FCoE VLAN Discovery, note there is no native vlan defined, so the native vlan will be vlan 1, then note that vlan 1 is NOT included in the switchport trunk allowed list

However, observe show int trunk:

n5k1-1# show int eth1/1 trunk

--------------------------------------------------------------------------------
Port          Native  Status        Port
              Vlan                  Channel
--------------------------------------------------------------------------------
Eth1/1        1       trunking      --
--------------------------------------------------------------------------------
Port          Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Eth1/1        5,30

--------------------------------------------------------------------------------
Port          Vlans Err-disabled on Trunk
--------------------------------------------------------------------------------
Eth1/1        none

--------------------------------------------------------------------------------
Port          STP Forwarding
--------------------------------------------------------------------------------
Eth1/1        5

--------------------------------------------------------------------------------
Port          Vlans in spanning tree forwarding state and not pruned
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
Port          Vlans Forwarding on FabricPath
--------------------------------------------------------------------------------
Eth1/1        none
n5k1-1# 


As you can see the native VLAN is shown, it's VLAN 1, do we see the devices MAC address in VLAN 1?:

n5k1-1# show mac address-table vlan 1
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link
   VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------


Nope we do not, so we know it's not "really" a member of VLAN 1,

However, when we look at a wireshark capture on this interface, you will notice that CDP, LLDP and other certain traffic goes through, this is because when you don't allow the native VLAN, it only does not allow unicast traffic, other traffic is allowed, including guess what? FIP VLAN Discovery


Here is our interface:


n5k1-1# show run int vfc2

!Command: show running-config interface vfc2
!Time: Mon Jun 10 03:54:09 2013

version 5.2(1)N1(4)

interface vfc2
  bind interface Ethernet1/1
  switchport trunk allowed vsan 30
  no shutdown
Here is the relevant VSAN Config:


vsan database
  vsan 30
  vsan 30 interface vfc2


and of course, the FCoE VLAN is defined

vlan 30
  fcoe vsan 30



Ok? Good

So let's look and see what our vfc2 interface is doing:

n5k1-1# show int vfc2
vfc2 is trunking
    Bound interface is Ethernet1/1
    Hardware is Ethernet
    Port WWN is 20:01:00:05:73:cd:73:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 30
    Trunk vsans (admin allowed and active) (30)
    Trunk vsans (up)                       (30)
    Trunk vsans (isolated)                 ()


It's logged in, the FCoE VLAN was discovered successfully even without the native vlan being allowed, just for a laugh I will reset the switch to make sure it works:


    Bound interface is Ethernet1/1
    Hardware is Ethernet
    Port WWN is 20:01:00:05:73:cd:73:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 30
    Trunk vsans (admin allowed and active) (30)
    Trunk vsans (up)                       (30)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()
    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      0 frames input, 0 bytes
        0 discards, 0 errors
      0 frames output, 0 bytes
        0 discards, 0 errors
    last clearing of "show interface" counters Mon Jun 10 04:10:47 2013

    Interface last changed at Mon Jun 10 04:10:51 2013

Still works, next i shut down the LAN on the switchport, this should disable all non-FCoE VLAN's on the interface, for good measure i will even change the VSAN to VSAN 10 and VLAN 10 as the FCoE VLAN




n5k1-1(config-if)# show run int eth1/1

!Command: show running-config interface Ethernet1/1
!Time: Mon Jun 10 04:13:52 2013

version 5.2(1)N1(4)

interface Ethernet1/1
  switchport mode trunk
  switchport trunk allowed vlan 10
  spanning-tree port type edge trunk
  shutdown lan

VFC Interface:
interface vfc2
  bind interface Ethernet1/1
  switchport trunk allowed vsan 10
  no shutdown

VSAN Database:
vsan database
 vsan 10 interface vfc2


Guess what? even with shutdown LAN, it worked fine:


n5k1-1(config-if)# show int vfc2
vfc2 is trunking
    Bound interface is Ethernet1/1
    Hardware is Ethernet
    Port WWN is 20:01:00:05:73:cd:73:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 10
    Trunk vsans (admin allowed and active) (10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()
    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec



My hypothesis is that the FIP VLAN discovery protocol works over the native VLAN like CDP, so even if you don't allow it expitically, because its a network protocol its allowed anyway.

THE MAIN THING TO TAKE AWAY FROM THIS IS: If you have FIP VLAN Discovery supported on your adapter, no worries, use it, otherwise you must specify manually, regardless of if you are specifying manually or not, you don't have to include the native VLAN in your switchport trunk allowed list, although you can if you want to and it shouldn't hurt anything