CCIE DC: Advanced FCoE

Hi Guys!



In this blog post I am going to look at quite advanced FCoE, this article assumes you already know the basics of FCoE, What a VE Port is, the "basic" process that happens when an FCoE device comes online, etc etc.

We are going to dive deeper into the process of an FCoE Device logging in, and we are going to investigate the topologies we support and that we could potentially see in the CCIE DC based on the versions specified in the blueprint.

Please note that this blog post was written while studying on live equipment, so some of the points made etc may be a little disjointed.

OK, My first piece of advice is: Let's say you have an N5K, and you want to put it into NPV mode, always use "feature fcoe-npv", this mode does not require a restart AND offers you all the same features as feature npv, except for end host ethernet mode, but really why do we need that on our 5k?

FCDOMAIN

Let's start right at the very top with FCDOMAIN, FCDomain controls which switches join the fabric and what switch-id they are assigned, the switch-id is key for FC as it allows traffic to be identified as going to a particular switch in the domain.

Now, you can specify the ID that the switch requests from the principal switch (which ultimately has authority over which domain ID's are allocated) using preferred or static, if you specify static, the switch will ONLY accept that allocated domain-id and will isolate itself if that domain-ID is already allocated.

If you say preferred, the switch will accept any FCID but prefers the domain-id you specify.

fcdomain domain 10 preferred vsan 10

if you are on a switch, and you want to have this actually take affect you need to do a restart, more importantly for preferred fcdomain's you need to do a DISRUPTIVE restart:

fcdomain restart disruptive vsan 10

the DISRUPTIVE keyword is hidden, keep that in mind!

There are two options in the fcdomain that might be a little confusing, commit and distribute, these are NOT required for anything else other than the distribution of the configuration of the "allowed"


fcdomain allowed 1-200 vsan 10

A super useful command to check if information is going to be distributed across the CFS is:

show cfs aplication
----------------------------------------------
 Application    Enabled   Scope
----------------------------------------------
 fwm            Yes       Physical-eth
 ntp            No        Physical-fc-ip
 stp            Yes       Physical-eth
 fscm           Yes       Physical-fc
 role           No        Physical-fc-ip
 rscn           No        Logical
 radius         No        Physical-fc-ip
 fctimer        No        Physical-fc
 syslogd        No        Physical-fc-ip
 fcdomain       Yes       Logical session-mgr    Yes       Physical-ip
 device-alias   Yes       Physical-fc

The above output shows because I entered the "Fcdomain distribute" command on BOTH switches, if I did not, the information would not distribute, you can also confirm that you have peers who are willing to distribute this info to each other:

n5k1# show cfs peers name fcdomain  vsan 10

Scope      : Logical [VSAN 10]
--------------------------------------------------------------------------------
 Domain Switch WWN              IP Address
--------------------------------------------------------------------------------
 10     20:00:54:7f:ee:b8:4c:40 10.9.1.15                               [Local]
                                n5k1
 20     20:00:54:7f:ee:b8:4b:40 10.11.1.15

Total number of entries = 2


The above output confirms that when i commit on one end, it will pass over to the other device.



n5k2(config)# fcdomain allowed 1-200 vsan 10
n5k2(config)# fcdomain commit vsan 10


Look on N5k1 now:
n5k1# show run | inc fcdomain
fcdomain distribute
fcdomain domain 10 preferred vsan 10
fcdomain allowed 1-200 vsan 10
fcdomain commit vsan 10


Next let's look at FCoE E Trunks


E Trunks are easy as pie to configure:

interface Ethernet1/10
  switchport mode trunk
  switchport trunk allowed vlan 1,10

 You must make sure you are allowing the appropriate FCoE VLAN and it CANNOT be native. Configure this on each end.

Next create the interface and bind it:


interface vfc10
  bind interface Ethernet1/10
  switchport mode E
  switchport trunk allowed vsan 10
  no shutdown

 Easy as that guy's, it's done now and ready to go!

n5k1# show int vfc10
vfc10 is trunking
    Bound interface is Ethernet1/10
    Hardware is Ethernet
    Port WWN is 20:09:54:7f:ee:b8:4c:7f
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Trunk vsans (admin allowed and active) (10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()
    1 minute input rate 144 bits/sec, 18 bytes/sec, 0 frames/sec
    1 minute output rate 104 bits/sec, 13 bytes/sec, 0 frames/sec
      1851 frames input, 209816 bytes
        0 discards, 0 errors
      1850 frames output, 226100 bytes
        0 discards, 0 errors
    last clearing of "show interface" counters Sat Jun 15 02:51:06 2013

    Interface last changed at Sat Jun 15 02:51:33 2013


Let's look even closer at some of the things that have been negotiated

n5k1# show lldp int eth1/10
Interface Information:
  Enable (tx/rx/dcbx): Y/Y/Y    Port Mac address: 54:7f:ee:b8:4c:51

You can tell from the above that DCBX has been negotiated and the below shows the negotiated queuing.


n5k1# show queuing interface eth1/10
Ethernet1/10 queuing information:
  TX Queuing
    qos-group  sched-type  oper-bandwidth
        0       WRR             50
        1       WRR             50

  RX Queuing
    qos-group 0
    q-size: 360960, HW MTU: 1500 (1500 configured)
    drop-type: drop, xon: 0, xoff: 360960
    Statistics:
        Pkts received over the port             : 1010
        Ucast pkts sent to the cross-bar        : 0
        Mcast pkts sent to the cross-bar        : 1010
        Ucast pkts received from the cross-bar  : 0
        Pkts sent to the port                   : 875
        Pkts discarded on ingress               : 0
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

 qos-group 1
    q-size: 79360, HW MTU: 2158 (2158 configured)
    drop-type: no-drop, xon: 20480, xoff: 40320
    Statistics:
        Pkts received over the port             : 2158
        Ucast pkts sent to the cross-bar        : 2158
        Mcast pkts sent to the cross-bar        : 0
        Ucast pkts received from the cross-bar  : 0
        Pkts sent to the port                   : 0
        Pkts discarded on ingress               : 0
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

  Total Multicast crossbar statistics:
    Mcast pkts received from the cross-bar      : 875


This is all so far relatively simple and straightforward, let's look at the exact config for a normal CNA port:

interface Ethernet1/1
  switchport mode trunk
  switchport trunk allowed vlan 1,10
  spanning-tree port type edge trunk

Here is the vFC configuration:


interface vfc1
  bind interface Ethernet1/1
  switchport trunk allowed vsan 10
  no shutdown


If we then go to the Adapter on the server and ensure it uses vlan 10 as the FCoE Vlan, the interface comes up no worries whatsoever:



n5k2# show int vfc1
vfc1 is trunking
    Bound interface is Ethernet1/1
    Hardware is Ethernet
    Port WWN is 20:00:54:7f:ee:b8:4b:7f
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 10
    Trunk vsans (admin allowed and active) (10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             ()
    1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
    1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
      19 frames input, 2144 bytes
        0 discards, 0 errors
      19 frames output, 2572 bytes
        0 discards, 0 errors
    last clearing of "show interface" counters Sat Jun 15 03:35:32 2013

    Interface last changed at Sat Jun 15 03:50:06 2013





Let's look at the final option, which is adapter FEX and vHBA's,


interfaceEthernet1/1
  switchport mode vntag
!
interface Vethernet1
  switchport mode trunk
  switchport trunk allowed vlan 10
  bind interface Ethernet1/1 channel 3
!


interface vfc1
  bind interface Vethernet1
  switchport trunk allowed vsan 10
  no shutdown
 

vsan database
  vsan 10 interface vfc1


This will bring up the vFC interface


Now, if your stuck in the dreaded, initlization state, here are some things to check:

• For F-Port's check to make sure that the VLAN is being tagged down the trunk
• Check to make sure that the VSAN has got the interface bound

Fabric-Provided MAC address

A fabric provided MAC address is used in FCoE by the nexus to allocate a MAC address to the ENode, which is the CNA and the adapter that actually sits on the FCoE Enabled VLAN,

These fabric provided Mac addresses are used by the FCoE Switch as the destination for FCoE traffic that it needs to send to a particular host, because FCoE remember is just ethernet, so it still has to use a source and destination mac address for it's FCoE Frames,

CNA's that login to the fabric can either provide there own FCoE MAC Address, or alternatively be allocated one via the switch

an FCF-MAC is the actual MAC address of the switch itself on the FCoE, so in other words this is the FCoE MAC address that the FCF will use as the source for traffic that it sends, and the CNA will use as the destination when it sends (because remember, FCoE is treated as point to point links so that things like spanning-tree etc don't need to be taken into account)

n5k1# show fcoe
Global FCF details
        FCF-MAC is 54:7f:ee:b8:4c:40
        FC-MAP is 0e:fc:00        FCF Priority is 128
        FKA Advertisement period for FCF is 8 seconds

VFC MAC details
        vfc1 FCF-MAC is 54:7f:ee:b8:4c:40
        vfc10 FCF-MAC is 54:7f:ee:b8:4c:51


Now, the FC-MAP command is a way to prevent cross-fabric talk, basically this value is by default set to 0e:fc:00, which by default would allow every Nexus 7k, 5k etc to talk to each other, if these values do not match and you attempt to trunk two switches together, they will not form properly as FCoE will discard all frames with a diffirent fc-map, this is to prevent a potential loop forming, you should change this value when you have two completely seperate fabrics and you want to ENSURE they never merge.

This value is initially exchanged in the FIP protocol messages

This can be changed with:

fcoe fcmap 0xefc01

The FCoE negotiation between two E ports will now never come up.



n5k1# show fcoe database

-------------------------------------------------------------------------------
INTERFACE       FCID            PORT NAME               MAC ADDRESS
-------------------------------------------------------------------------------
VE Ports:
-------------------------------------------------------------------------------
INTERFACE       MAC ADDRESS             VSAN
-------------------------------------------------------------------------------
vfc10           54:7f:ee:b8:4b:31       10
 



Let's look at the next topology, which is using a FEX connected port for FCoE, this will also give us chance to look at the FCF-Priority command


There is nothing special about configuring FCoE with a FEX either, shown below is the configuration for a single-attached FEX (it has to be a FEX that supports FCoE Obviously ;)


Here is an example of the FEX config:

feature fex
fex 101

interface Ethernet1/15
  switchport mode fex-fabric
  fex associate 101

You can then see that this FEX is associated:


n5k1# show fex detail
FEX: 101 Description: FEX0101   state: Online
  FEX version: 5.2(1)N1(1) [Switch version: 5.2(1)N1(1)]
  FEX Interim version: 5.2(1)N1(1)
  Switch Interim version: 5.2(1)N1(1)
  Module Sw Gen: 12594  [Switch Sw Gen: 21]
  post level: complete
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/15
  FCoE Admin: false
  FCoE Oper: true
  FCoE FEX AA Configured: false
  Fabric interface state:
    Eth1/15 - Interface Up. State: Active
  Fex Port        State  Fabric Port
       Eth101/1/1  Down        None
       Eth101/1/2  Down        None
       Eth101/1/3  Down        None


From here, your actual vfc config is exactly what you would expect:

 interface Ethernet101/1/20
  switchport mode trunk
  switchport trunk allowed vlan 1,10

interface vfc2
  bind interface Ethernet101/1/20
  switchport trunk allowed vsan 10
  no shutdown

As you can see this is nothing unusual, it's exactly the same FEX config you are used to, there is nothing weird about this config.

Here is the interface, up and happy:

n5k1# show int vfc2
vfc2 is trunking
    Bound interface is Ethernet101/1/20
    Hardware is Ethernet
    Port WWN is 20:01:54:7f:ee:b8:4c:7f
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 10
    Trunk vsans (admin allowed and active) (10)
    Trunk vsans (up)                       (10)



OK, next let's take a look at some Etherchannels, Let's start with the easiest which is an VE  SAN Port Channel:

interface Ethernet1/10
  switchport mode trunk
  switchport trunk allowed vlan 1,10
  channel-group 30 mode active

interface Ethernet1/20
  switchport mode trunk
  switchport trunk allowed vlan 1,10
  channel-group 30 mode active

interface port-channel30
  switchport mode trunk
  switchport trunk allowed vlan 1,10
  speed 10000








Next let's look at a fabric FEX, i.e. a FEX that is homed to two devices, config is not spelt out as this is an advanced FCoE Config but here is roughly the config for a FEX associated to two home devices:


interface Ethernet1/15
  switchport mode fex-fabric
  fex associate 101
  channel-group 2


interface port-channel2
  switchport mode fex-fabric
  fex associate 101
  vpc 2





If after configuring FEX that is dual homed to two 5k's, if you try and bind, you will see the following:

n5k1(config)# int vfc2
n5k1(config-if)# bind int eth101/1/20
ERROR: fcoe_mgr: Cannot bind the VFC as the Fex not fcoe configured (err_id 0x4207
005B)



n5k1(config-if)# no vpc
2013 Jun 15 13:05:44 n5k1 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_OFFLINE: FEX-101 Off-
2013 Jun 15 13:05:44 n5k1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is offline
n5k1(config-if)#
n5k1(config-if)# 2013 Jun 15 13:06:29 n5k1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101
 is online
2013 Jun 15 13:06:29 n5k1 %$ VDC-1 %$ %NOHMS-2-NOHMS_ENV_FEX_ONLINE: FEX-101 On-li
ne
2013 Jun 15 13:06:30 n5k1 %$ VDC-1 %$ %PFMA-2-FEX_STATUS: Fex 101 is online

n5k1(config-if)# int vfc2
n5k1(config-if)# bind int eth101/1/20

As you can see from the above however, if we are NOT using the FEX dual-home, we don't require the FCoE command.


This leads us into one more topology, which is a vPC enabled FCoE, Basically this is perfectly fine as long as you keep the two fabrics seperate, and you can only have ONE interface going down from each vPC switch

My lab topology i could not test this, so please see the link below


http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_fcoe_ops_appC.html


So the next topology would be an F-Port Channel, you really can't do this to a host, the only time you would do a f-port channel would be from an NPV switch up to a NPIV switch as we will demonstrate, here is a topology diagram:







Let's take a quick look at the config, and while we are here we will probably spend a little time on FCoE-NPV as well

Here is our config on our NPV Switch:

vlan 10
  fcoe vsan 10


interface port-channel1
  switchport mode trunk
  switchport trunk allowed vlan 10
  speed 10000

interface vfc1
  bind interface port-channel1
  switchport mode NP
  switchport trunk allowed vsan 10
  switchport trunk mode on
  no shutdown
!

This config works as you would expect and comes up just fine, the only thing during my testing i could nto get working was multiple VSAN's up this link, however i suspect this is a limitation  of FCoE NPV mode

UPDATE: This does work, you can have VFC with multiple VSAN's, the trick is they won't come up across the link until you actually have interfaces (F Ports) in that particular VSAN, Thanks to the CCIE DC Facebook group for this info!



switch(config-if)# show int vfc1
vfc1 is trunking
    Bound interface is port-channel1
    Hardware is Ethernet
    Port WWN is 20:00:00:05:73:cd:73:bf
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 1
    Trunk vsans (admin allowed and active) (10,20)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             (20)


If I add a F Port downstream from this NPV switch into VSAN 20, the NPV will allow me to have multiple vlan's going up the trunk:


n5k1-2(config)# show int vfc100
vfc100 is trunking (Not all VSANs UP on the trunk)
    Bound interface is port-channel100
    Hardware is Ethernet
    Port WWN is 20:63:00:05:73:ca:06:7f
    Admin port mode is F, trunk mode is on
    snmp link state traps are enabled
    Port mode is TF
    Port vsan is 1
    Trunk vsans (admin allowed and active) (1,10,20)
    Trunk vsans (up)                       (10,20)    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             (1)
    1 minute input rate 8 bits/sec, 1 bytes/sec, 0 fr




Next, let's look at the NPV traffic map



N5k2(config)# show npv status

npiv is disabled

disruptive load balancing is disabled

External Interfaces:
====================
  Interface:   vfc1, State: Trunking
        VSAN:    1, State: Waiting For VSAN Up
        VSAN:   10, State: Up, FCID: 0x720000
        VSAN:   20, State: Up

  Number of External Interfaces: 1

Server Interfaces:
==================

  Number of Server Interfaces: 0

The disruptive command is potentially dangerous, basically if a new link is added between two switches from an NPV point of view, so say for example in our diagram we where not using a port channel and instead where just using two links between our NPV and our NPIV switch, and then added a third link,  npv disruptive would immediately logout some attached servers and re-attach them to this new link so as to distribute the load across the new uplink (now of course, if you where using a port channel, you could avoid this problem :))


N5k2(config)# npv auto-load-balance ?
  disruptive  Enable disruptive auto load balancing among external links

You can force traffic for a particular server to go out a particular link by configuring the following:

N5k2(config)# npv traffic-map server-interface vfc2 external-interface vfc1
N5k2(config)# show npv traffic-map

NPV Traffic Map Information:
----------------------------------------
Server-If       External-If(s)

----------------------------------------
vfc2            vfc1
----------------------------------------
N5k2(config)# 




FCF Priority


Let's quickly talk about FCoE FCF-Priority, mostly because it's something you don't really have to worry about as it would be very difficult to test on

Let's take a look at the diagram below, which i SHAMELESSLY stole from brasstacksblog and there excellent article on FC and FIP below, I hope the author of the blog does not mind. If you do please leave a comment and I will remove the picture immediately :)






http://brasstacksblog.typepad.com/brass-tacks/2010/11/fip-fip-snooping-bridges-and-fcfs-part-1-fip-the-fcoe-initialization-protocol.html

Ok so check out the picture above, in the picture above, we can see the "lossless ethernet switch", this is basically just a switch that supports something called "FIP snooping" which means that while that particular switch itself is NOT an FCF forwarding it can forward the FIP frames, now in this picture there are two switches that ARE FCF Forwarders (as shown by the yellow boxes), each of these has a priority, why? bare in mind that FCoE uses a point to point topology, so in our weird topology shown above the CNA can only log into ONE FCF, it cannot log into both, so the FCF Priority command allows the CNA to pick the correct FCF to login to.

If you took the Lossless ethernet switch in the picture above completely out of the equation, and instead had a CNA with two ports, each port going to each seperate switch, the FCF priority command would not have any effect, the CNA would login to both switches on each seperate port and establish a PTP link on each FCF, so as you can see the FCF Priority command really is for very esoteric layouts.





Load Balancing
(Editors note: I tested All of this using show int eth1/10 | inc rate and other similiar commands, I can verify that the below is 100 percent true based on my own lab testing)


Let's say you just have two FC Links between switches, as you may know, the traffic will be load balanced across these links based on a variety of attributes. The FSPF load balancing for the VSAN
controls what FSPF is using to load balance:


n5k1# show vsan
vsan 1 information
         name:VSAN0001  state:active
         interoperability mode:default
         loadbalancing:src-id/dst-id/oxid
         operational state:down

vsan 10 information
         name:VSAN0010  state:active
         interoperability mode:default
         loadbalancing:src-id/dst-id/oxid
         operational state:up

n5k1(config-vsan-db)# vsan 10 loadbalancing ?
  src-dst-id     Src-id/dst-id for loadbalancing
  src-dst-ox-id  Ox-id/src-id/dst-id for loadbalancing(Default)
This is a global setting for that particular VSAN, note that it will ALSO affect the load balancing of your SAN-Port-Channels, i.e. if you leave it as the default, it will load balance even san-port-channel traffic based on ox-id, this applies for both FSPF _and_ SAN Port Channels


It also applies if the E Ports are VE Ports, so even then the usual rules of FSPF are obeyed.

Remember, Both these settings are on a per-switch basis! you _CAN_ mismatch them between switches and one switch will load balance one way, one switch will load balance the other


However, for your FCoE port channels, how do you get them to include the OX-ID?

Let me tell you something: I couldn't find this documented anywhere, i could find NO WHERE on cisco where it said, but once i knew the command, i found this ask the expert post:


https://supportforums.cisco.com/thread/2134134

"On the Nexus 5000, the default load balancing mechanism on the LACP port-channel is source-destination. If we leave it in this state, all the FCoE traffic will take the same link in the port channel when the Nexus 5000 is forwarding frames to the upstream device.

To enable the Nexus 5000 to load balance using exchange IDs, we configure it for 'source-dest-port' load balancing.

Nexus5000(config)# port-channel load-balance ethernet ?
  destination-ip    Destination IP address
  destination-mac   Destination MAC address
  destination-port  Destination TCP/UDP port
  source-dest-ip    Source & Destination IP address   --------> SID/DID
  source-dest-mac   Source & Destination MAC address
  source-dest-port  Source & Destination TCP/UDP port --------> SID/DID/OXID
  source-ip         Source IP address
  source-mac        Source MAC address
  source-port       Source TCP/UDP port
 "


This double confirms it, i have also tested it myself, so in conclusion, if you have an FCoE Port channel and you want it to load balance based on OX-ID, you need to include the source-dest port

YOU MUST DO THIS ON BOTH SIDES