CCIE DC: Advanced FCoE Part 3

Hi Guys

Final blog post on FCoE and then Cisco I have learnt this as best I can! so if it's still not enough I am not sure what to do ha ha ha,

OK let's Roll.

So First of all, N7K + FCoE

We all know you need a separate storage VDC, this introduces some... "interesting" configuration

The first thing you must do, is install the FCoE Feature set on the root VDC:

 switch(config)# install feature-set fcoe


Next, you must individually license your line card for FCoE, determine which is your F line-card using the show module command:

switch(config)# show module
Mod  Ports  Module-Type                         Model              Status
---  -----  ----------------------------------- ------------------ ----------
3    32     10 Gbps Ethernet XL Module          N7K-M132XP-12L     powered-up
4    32     1/10 Gbps Ethernet Module           N7K-F132XP-15      powered-up6    0      Supervisor module-1X                N7K-SUP1           active *



switch(config)# license fcoe module 4

Next, you need to create a FCoE VLAN that will be used by your VDC and of course allocate interfaces

vdc SW1-5 id 2 type storage
  allow feature-set fcoe
  allocate interface Ethernet4/1-2
 allocate fcoe-vlan-range 2000 from vdcs switch (switch is the default VDC)
!
 

 You also need to set the QoS Policy, let's see what happens if we don't

You can configure all your FCoE on your 7k like you normally would:

switch-SW1-5(config-if)# feature lldp

switch-SW1-5(config-if)# feature-set fcoe

switch-SW1-5(config-if)# show run | sect vsan|vlan|4/1|vfc
vlan 1
vlan 2000
  fcoe vsan 10 
vsan database
  vsan 10
interface vfc1
  bind interface Ethernet4/1
interface Ethernet4/1
  switchport mode trunk
  switchport trunk allowed vlan 2000

When you bring up the interface on the other end, you will receive the following log message:

switch-SW1-5(config-if)# 2013 Jun 18 12:30:48 switch-SW1-5 %$ VDC-2 %$ %IPQOSMGR-2-QOSMGR_DCBXP_PFC_CMP_FAIL_MSG:  Ethernet4/1 - qos config 'Priority-flow-control' not compatible with the peer

So you need to fix this, as you know you can't do this on the Storage VDC

switch-SW1-5(config)# system qos
ERROR: network-qos configs can only be done from default VDC


So return to the default VDC and implement a FCoE Storage policy


switch-SW1-5# show policy-map type network-qos

  Type network-qos policy-maps
  ============================
  policy-map type network-qos default-nq-4e-policy
    class type network-qos c-nq-4e-drop
      congestion-control tail-drop
      mtu 1500
    class type network-qos c-nq-4e-ndrop-fcoe
      pause
      mtu 2112
    class type network-qos c-nq-4e-ndrop
      pause
      mtu 2112
  policy-map type network-qos default-nq-6e-policy
    class type network-qos c-nq-6e-drop
      congestion-control tail-drop
      mtu 1500
    class type network-qos c-nq-6e-ndrop-fcoe
      pause
      mtu 2112
    class type network-qos c-nq-6e-ndrop
      pause
      mtu 2112
  policy-map type network-qos default-nq-7e-policy
    class type network-qos c-nq-7e-drop
      congestion-control tail-drop
      mtu 1500
    class type network-qos c-nq-7e-ndrop-fcoe
      pause
      mtu 2112
  policy-map type network-qos default-nq-8e-policy
    class type network-qos c-nq-8e
      congestion-control tail-drop
      mtu 1500





You can pick any of the above your happy to use as long as the policy has a FCoE class in it, we will use policy default-nq-7e-policy

switch(config)# system qos

switch(config-sys-qos)# service-policy type network-qos default-nq-7e-policy
switch(config-sys-qos)# 2013 Jun 18 12:34:21 switch %$ VDC-1 %$ %IPQOSMGR-2-QOSMGR_NETWORK_QOS_POLICY_CHANGE: Policy default-nq-7e-policy is now active
2013 Jun 18 12:34:21 switch-SW1-5 %$ VDC-2 %$ %IPQOSMGR-2-QOSMGR_NETWORK_QOS_POLICY_CHANGE: Policy default-nq-7e-policy is now active
 

Once this is fixed up, your FCoE Storage will come up just fine:

switch-SW1-5# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
    Bound interface is Ethernet4/1
    Hardware is Ethernet
    Port WWN is 20:00:64:a0:e7:3f:b4:bf
    Admin port mode is E, trunk mode is on
    snmp link state traps are enabled
    Port mode is TE
    Port vsan is 1
    Speed is 10 Gbps
    Trunk vsans (admin allowed and active) (1,10)
    Trunk vsans (up)                       (10)
    Trunk vsans (isolated)                 ()
    Trunk vsans (initializing)             (1)
    211 fcoe in packets
    23464 fcoe in octets
    211 fcoe out packets
    22940 fcoe out octets
    Interface last changed at Tue Jun 18 12:35:19 2013

Hooray!


Shared Interface

In the example above the interface is a dedicated interface for the FCoE Storage, obviously if we are going down to our servers or maybe its an important trunk port, this won't be the case, so we have to be able to share the FCoE Interface (hey That's why its converged right?!)

So let's look at that
First of all we need to remove that interface from the storage VDC since we allocated it as dedicated before, only way to do that is allocate it to anotehr VDC


switch(config)# vdc switch
switch(config-vdc)# allocate interface eth4/1


Next, let's see what happens if we try and allocate it in our storage VDC:

 switch(config-vdc)# allocate shared interface eth4/1
Command execution failed due to: Configuration cannot be done for this port mode
                                                   ^

Huh? Port Mode? yep, it has to be a trunk interface:

switch(config)# int eth4/1
switch(config-if)# switchport mode trunk

Now we can allocate it (or can we?)

switch(config-if)# vdc SW1-5
switch(config-vdc)# allocate shared interface eth4/1
Command execution failed due to: Configuration cannot be done for this port mode
                                                   ^
Interface verification internal error at '^' marker.
 
switch(config-vdc)# show run int eth4/1

!Command: show running-config interface Ethernet4/1
!Time: Tue Jun 18 12:56:14 2013

version 6.0(2)

interface Ethernet4/1
  shutdown
  switchport mode trunk
  switchport trunk allowed vlan 2000
  spanning-tree port type edge trunk




We still can't allocate it? Why not? Its a trunk port etc, what is going on?

The issue is, remember your allocating an interface, just _ONE_ interface.. but think about it for a minute.. you can't allocate just one int since two interfaces are a member of a port group.. that's right, you need to set the port-mode for eth4/2 as well! Because even though your not explitically adding that one, the command will add it anyway:

switch(config-vdc)# int eth4/2
switch(config-if)# switchport mode trunk 


switch(config-if)# vdc SW1-5
switch(config-vdc)# allocate shared int eth4/1
Entire port-group is not present in the command. Missing ports will be included automatically
Ports that share the port group of the interfaces you have specified will be affected as well. Continue (y/n)? [yes] yes
switch(config-vdc)#

Yes! Now we have the shared interface allocated.

However, check this out:

switch-SW1-5(config-if)# int vfc10
switch-SW1-5(config-if)# bind int eth4/1
switch-SW1-5(config-if)# no shut
switch-SW1-5(config-if)# switchport mode e
vfc10: (error) configuration of this port mode not allowed
switch-SW1-5(config-if)# switchport ?


So, F Port mode ONLY supported if your doing FCoE and you want to use a shared interface, makes sense from Cisco point of view: they want you to use a shared int only if doing it down to a CNA, not shared ints for normal ethernet traffic.

You can verify with the show vdc shared membership command


switch-SW1-5(config)# show vdc shared membership

vdc_id: 2 vdc_name: SW1-5 interfaces:
        Ethernet4/1           Ethernet4/2          


OK I only have one more FCoE Thing then that is it for me for FCoE :p