CCIE DC: iSLB

Hi Guys

In this blog post I am going to go over iSLB, which is a protocol used with iSCSI to allow a degree of load balancing between multiple switches, it is fairly cool and relatively easy to configure so Let's get to it!

First of all, as always credit where it is due, I have got most of the info I am about to present you from: http://www.cisco.com/en/US/docs/switches/datacenter/mds9000/sw/6_2/configuration/guides/ip_services/nx-os/ciscsi.html#wp2998584

For those of you wondering where I practice all of this stuff, I practice it on two Cisco MDS 9216i device's I have at home. They where not very expensive to obtain around 1000 dollars for two and they have been really invaluable. I would suggest if your deadly serious about getting more time with storage that these are a good investment. Be forewarned: they are LOUD, as loud as a Juniper EX3200 (You heard those things? for a bloody "enterprise" class switch, they're stupidly ear-peircingly loud)

Anyway enough Juniper bashing, let's get to it!

OK first of all you still need to enable iSCSI just like you normally do:

Switch1(config)# feature iscsi

Now we are going to skip a head with iSLB a bit here so bare with me:

iSLB is all about load balancing, to load balance we normally require two switches or two devices right ;) or at least two ports, but in our example we are going to use two switches, so we need to make sure the config is distributed, it is strongly recommended to get your CFS distribution of the islb config going BEFORE you worry about configuring it up.

So let's do that!
Ok, enable islb CFS distribution:

(both switches)

Switch1(config)# islb distribute

Check that the two switches can see each other as peers:

Switch2# show cfs peers name islb

Scope      : Physical-fc
-------------------------------------------------------------------------
 Switch WWN              IP Address
-------------------------------------------------------------------------
 20:00:00:0d:ec:2b:3c:40 10.0.0.33                               [Local]
                         Switch2                                
 20:00:00:0d:ec:2d:4f:40 10.0.0.32           


Let's turn on the usual iSCSI stuff we need:
(Both switches)

Switch2(config)# iscsi enable module 1

Switch2(config)# int iscsi1/1
Switch2(config-if)# no shut


 
Awesome, they can. As you can see from: Scope: Physical-FC this particular distribution works with FC.

Ok next's lets configure the other aspects.

So you know when you configure a iSCSI initiator right? You put all the details in, you give it a static pwwn, you give it a static nwwn which can be system-assigned (recommended) or manually assigned, you tell it what VSAN it's going to be in, yadda yadda yadda.

Well, all of that pretty much remains the same with iSLB:


Switch2(config)# islb initiator name iqn.1991-05.com.microsoft:nervmainpc
Switch2(config-islb-init)# ?
  metric       Assign load balancing metric for the initiator  mutual-chap  Assign username for initiator's challenge (mutual chap)
  no           Negate a command or set its defaults
  static       Create static nWWN or pWWN for the initiator
  target       Configure iSLB initiator target  username     Assign username for iSLB login authentication
  vsan         Assign VSAN membership for the initiator
  zonename     Assign zone name for the initiator targets  end          Go to exec mode
  exit         Exit from command interpreter
  pop          Pop mode from stack or restore from name
  push         Push current mode to stack or save it under name
  where        Shows the cli context you are in


Let's quickly get the boring stuff that we already know how to do thanks to my previous article on iSCSI and concentrate on the new bits

Switch2(config-islb-init)# static pwWN system-assign 1
Switch2(config-islb-init)# static nWWN system-assign


Ok now let's look at these three new options

metric, target and zonename.

So when you do iscsi zoning, we talked about in my previous blog post that you have to do your iscsi access control, AND you gotta remember to do the FC zoning too! Well iSLB introduces a concept called "auto zoning" which allows iSCSI to automatically create the appropriate zoning info for you! Pretty cool hey?

But in order to do it, you need to tell the iscsi initiator what target it will be talking to, hence the target keyword:

Switch2(config-islb-init)# target ?
  device-alias  Device-alias of the fc-target
  pwwn          PWWN of the fc-target

Switch2(config-islb-init)# target pwwn 50:01:10:a0:00:18:31:80

You can specify a pwwn or a device-alias, Hooray!

Ok next up,is the zone name, this allows you to decide what the zone will be called when it is autocreated, otherwise the MDS in it's infinite wisdom will just choose one of it's own liking.



Switch2(config-islb-init)# zonename TheTwilightZone

Alright! Let's commit our change

Switch2(config)# islb commit


NOW, what step have we missed? Let' me give you a hint, look at the following output:


Switch2# show iscsi global
iSCSI/iSLB Global information (fabric-wide)
  Authentication: CHAP, NONE
  Initiator idle timeout: 300 seconds
  Dynamic Initiator: iSCSI
  iSLB Distribute: Enabled
  iSLB CFS Session: Does not exist
  Number of load balanced VRRP groups: 0
  Number of load-balanced initiators: 0
iSCSI/iSLB Global information (local to this switch)
  Import FC Target: Disabled
  Initiator Plogi timeout: 2 seconds
  Number of target node: 0
  Number of portals: 4
  Number of session: 0
  Failed session: 0, Last failed initiator name:


What have we failed to do so far?

We haven't actually configured an iSCSI target, and we haven't configured dynamic import either (shown above with the "Import FC Targed: Disabled")

So let's for the sake of ease for now, just import the target, we will configure a static target later.



Switch2(config)# iscsi import target fc 

Now the next thing we need to do, is enable the interface for VRRP, ISLB will NOT come up without VRRP, you will see an error message like so when you perform a show islb status

Switch1# show islb status
  iSLB Distribute: Enabled
  iSLB CFS Session: Does not exist
  Number of load balanced VRRP groups: 0
  Number of load-balanced initiators: 0


So configure VRRP on the interface

Switch1# show run int gi1/1

!Command: show running-config interface GigabitEthernet1/1
!Time: Sun Sep  6 09:11:29 1981

version 4.2(9)

interface GigabitEthernet1/1
  ip address 10.0.0.32 255.255.255.0
  no shutdown
  vrrp 1
    address 10.0.0.36
    no shutdown


Now you will get the magic output:

Switch1# show islb status
  iSLB Distribute: Enabled
  iSLB CFS Session: Exists (Initiated locally)
  Number of load balanced VRRP groups: 0
  Number of load-balanced initiators: 0


Let's now initiate to it with our iSCSI initiator:



Woo Hoo! It's done, what happened with the zoning stuff?

zone name switch1 vsan 1
  pwwn 50:01:10:a0:00:18:31:82
  pwwn 21:02:00:0d:ec:2d:4f:42
* fcid 0x020000 [pwwn 50:01:10:a0:00:18:31:80]

zone name ips_zone_TheTwilightZone vsan 1* fcid 0x010005 [symbolic-nodename iqn.1991-05.com.microsoft:nervmainpc]
* fcid 0x020000 [pwwn 50:01:10:a0:00:18:31:80]


It auto-created a zone for us! this is in our show zone active command, so it's not actually in our running config database.

Now our initiator has a IQN and we are good, the best bet at this point is to save your initiator:

Switch2(config)# iscsi save-initiator

OK looks pretty good so far, let's check out a few commands:




Switch1# show islb session detail
Initiator iqn.1991-05.com.microsoft:nervmainpc
  Initiator ip addr (s): 10.0.0.83
  Session #1 (index 2)
    Target iqn.1987-05.com.cisco:05.86f7c3042928b6638b93897dd6ab1d7b.500110a0001
83180
    VSAN 1, ISID 400001370000, TSIH 4097, Status active, no reservation
    Type Normal, ExpCmdSN 11, MaxCmdSN 138, Barrier 0
    MaxBurstSize 262144, MaxConn 1, DataPDUInOrder Yes
    DataSeqInOrder Yes, InitialR2T No, ImmediateData Yes
    Registered LUN 0, Mapped LUN 0
    Stats:
      PDU: Command: 10, Response: 10
      Bytes: TX: 324, RX: 0
    Number of connection: 1
    Connection #1 (index 1)
      Local IP address: 10.0.0.36, Peer IP address: 10.0.0.83
      CID 1, State: Full-Feature
      StatSN 12, ExpStatSN 0
      MaxRecvDSLength 65536, our_MaxRecvDSLength 262144
      CSG 3, NSG 3, min_pdu_size 48 (w/ data 48)
      AuthMethod none, HeaderDigest None (len 0), DataDigest None (len 0)
      Version Min: 0, Max: 0
      FC target: Up, Reorder PDU: No, Marker send: No (int 0)
      Received MaxRecvDSLen key: Yes
      Stats:
        Bytes: TX: 324, RX: 0


We get some pretty good info out of this!

OK but so far we haven't really done anything exciting, yep we have ISLB but we haven't actually split it across the two locations yet, we need to see that working!


So let's configure the second switch with appropriate VRRP configuration:
interface GigabitEthernet1/1
  ip address 10.0.0.33 255.255.255.0
  no shutdown
  vrrp 1
    address 10.0.0.36
    no shutdown

Now we have VRRP


Switch2# show vrrp
      Interface  VR IpVersion Pri   Time Pre State   VR IP addr
---------------------------------------------------------------------------
        GigE1/1   1   IPv4    100    1 s     backup  10.0.0.36


We need to show that we intend to use this VRRP Group for load balancing ISLB:

Switch2(config)# islb vrrp 1 load-balance

Commit the change:

Switch2(config)# islb commit

We NEXT HAVE TO CONFIGURE AN ISCSI INT ON THE SECOND SWITCH:


If you don't do this, the iSLB will not load balance across the two switches, so this is critically important.

 Switch2(config)# int iscsi1/1
Switch2(config-if)# no shut

This now shows up in our SLB config:

 Switch2(config)# show islb vrrp
-- Groups For Load Balance --

    VRRP group id 1
        Address type: IPv4
        Configured status: Enabled

-- Interfaces For Load Balance --

    Interface GigabitEthernet1/1
        Switch wwn: 20:00:00:0d:ec:2b:3c:40
        VRRP group id: 1, VRRP IP address: 10.0.0.36
            Interface VRRP state: backup
            Interface load: 0
            Interface redirection: enabled
            Group redirection: enabled
        Number of physical IP address: 1
            (1) 10.0.0.33
        Port vsan: 1
        Forwarding mode: store-and-forward
        Proxy initiator mode: disabled
        iSCSI authentication: CHAP or None

    Interface GigabitEthernet1/1
        Switch wwn: 20:00:00:0d:ec:2d:4f:40
        VRRP group id: 1, VRRP IP address: 10.0.0.36
            Interface VRRP state: master
            Interface load: 0
            Interface redirection: enabled
            Group redirection: enabled
        Number of physical IP address: 1
            (1) 10.0.0.32
        Port vsan: 1
        Forwarding mode: store-and-forward
        Proxy initiator mode: disabled
        iSCSI authentication: CHAP or None

-- Initiator To Interface Assignment --


Very cool

Switch2(config)# show islb status
  iSLB Distribute: Enabled
  iSLB CFS Session: Does not exist - This simply means there's no active CFS session, NOT that the CFS session is not working.
  Number of load balanced VRRP groups: 1
  Number of load-balanced initiators: 0


OK let's see this go:

Switch2# show islb vrrp summary

                         -- Groups For Load Balance --
--------------------------------------------------------------------------------
               VR Id             VRRP Address Type             Configured Status
--------------------------------------------------------------------------------
                   1                          IPv4                       Enabled

                       -- Interfaces For Load Balance --
--------------------------------------------------------------------------------
                                                             Initiator  Redirect
 VR Id         VRRP IP              Switch WWN     Interface      Load   Enabled
--------------------------------------------------------------------------------
     1       10.0.0.36 20:00:00:0d:ec:2b:3c:40       GigE1/1      1000       Yes
M    1       10.0.0.36 20:00:00:0d:ec:2d:4f:40       GigE1/1         0       Yes

                    -- Initiator To Interface Assignment --
--------------------------------------------------------------------------------
Initiator  VR Id         VRRP IP              Switch WWN               Interface
--------------------------------------------------------------------------------
iqn.1991-05.com.microsoft:nervmainpc
               1       10.0.0.36 20:00:00:0d:ec:2b:3c:40      GigabitEthernet1/1

There is a lot going on in this output, so let's take a look

First of all, you can see that their are two VRRP iSLB switches, one of them has an "M" next to it, this is the VRRP master, the VRRP master performs the iSCSI login Redirect Messaging and thus is under slightly higher load than the other switches, therefore the first login in this instance is redirected to the other switch (Switch2), the non-VRRP master, to try and even up the load a little bit.

Let's add another initiator:

First, we need to start the initiator trying to connect, this will give us the name we can then use to create the initiator

 Switch1(config)# islb initiator name iqn.1991-05.com.microsoft:01.dat.com
Switch1(config-islb-init)# static pWWN system-assign 2
Switch1(config-islb-init)# static nwWN system-assign
Switch1(config-islb-init)# target pwwn 50:01:10:a0:00:18:31:80
Switch1(config-islb-init)# exit


Notice this time i have not specified a zone name, next we distribute the config:

Switch1(config)# islb commit

Done, let's see what we have now



Switch1# show islb vrrp summary

                         -- Groups For Load Balance --
--------------------------------------------------------------------------------
               VR Id             VRRP Address Type             Configured Status
--------------------------------------------------------------------------------
                   1                          IPv4                       Enabled

                       -- Interfaces For Load Balance --
--------------------------------------------------------------------------------
                                                             Initiator  Redirect
 VR Id         VRRP IP              Switch WWN     Interface      Load   Enabled
--------------------------------------------------------------------------------
     1       10.0.0.36 20:00:00:0d:ec:2b:3c:40       GigE1/1      1000       Yes
M    1       10.0.0.36 20:00:00:0d:ec:2d:4f:40       GigE1/1      1000       Yes

                    -- Initiator To Interface Assignment --
--------------------------------------------------------------------------------
Initiator  VR Id         VRRP IP              Switch WWN               Interface
--------------------------------------------------------------------------------
iqn.1991-05.com.microsoft:010-1.dat.com
               1       10.0.0.36 20:00:00:0d:ec:2d:4f:40      GigabitEthernet1/1
iqn.1991-05.com.microsoft:nervmainpc
               1       10.0.0.36 20:00:00:0d:ec:2b:3c:40      GigabitEthernet1/1



Success! The login has been distributed between the two MDS Switches!

I promised I was going to talk about one more attribute for iSLB initiators, which is the "metric" keyword.

This allows us to set a higher "load" cost for each initiator, so for example, if we have a database server that we know is going to generate a high number of IOPS, we can tell the MDS that this one should be considered as quite costly from a load perspective, so it will load up the interface carrying this database less.

Switch1(config)# islb initiator name iqn.1991-05.com.microsoft:nervmainpc

Switch1(config-islb-init)# metric 5000


OK pretty clear so far with iSLB I hope guys! I hope this has shown you a bit, let's just quickly go over one more aspect.

You can configure a virtual-target with iSLB just like you can with iSCSI, there are no new options to worry about here :)

Switch1(config)# islb virtual-target name iqn.1998-08.test.com
Switch1(config-islb-tgt)# ?
  all-initiator-permit  Allow all iSLB initiator access to this target
  initiator             Allow iSLB initiator access to this target
  no                    Negate a command or set its defaults
  pWWN                  Enter the pWWN of the fc-target
  revert-primary-port   Revert back to primary port when it comes back up
  trespass              Enable Trespass Support
  end                   Go to exec mode
  exit                  Exit from command interpreter
  pop                   Pop mode from stack or restore from name
  push                  Push current mode to stack or save it under name
  where                 Shows the cli context you are in


I hope that helps someone out there!