CCIE DC: Troubleshooting iSCSI for CCIE DC

Hi Guys

After just finishing watching the excellent INE Storage videos, I thought that the coverage on iSCSI could use some expansion, so I have quickly run up this blog to go through a few of the better iSCSI troubleshooting tools available to you.

Here is one of the first one's I discovered:

debug ips iscsi msgs


This awesome command helps you really drill down and see what the issue with iSCSI is when your not able to find any targets in Windows:



I couldn't at first work out what was going on, my config is shown below:

feature iscsi
iscsi enable module 1
  vsan 50 interface iscsi1/1
interface iscsi1/1
  no shutdown
!

Any idea what is wrong with the above? My zoning? Let's check that out

SANB# show zone active vsan 50
zone name ISCSI vsan 50
* fcid 0x0f0002 [symbolic-nodename iqn.1991-05.com.microsoft:nervmainpc]
* fcid 0xcd0000 [pwwn 50:01:10:a0:00:18:31:80] [Target_Port1]
* fcid 0x0f0000 [pwwn 50:01:10:a0:00:18:31:82] [Target_port2]

zone name FC vsan 50
* fcid 0x0f0000 [pwwn 50:01:10:a0:00:18:31:82] [Target_port2]
* fcid 0x0f0001 [pwwn 50:01:10:a0:00:18:31:e6] [Server1_Port2]

The asterisk next to the Fcid indicates that I have the correct zoning, let's check out show iscsi global:

SANB# show iscsi global
iSCSI/iSLB Global information (fabric-wide)
  Authentication: CHAP, NONE
  Initiator idle timeout: 300 seconds
  Dynamic Initiator: iSCSI
  iSLB Distribute: Disabled
  iSLB CFS Session: Does not exist
  Number of load balanced VRRP groups: 0
  Number of load-balanced initiators: 0
iSCSI/iSLB Global information (local to this switch)
  Import FC Target: Disabled
  Initiator Plogi timeout: 2 seconds
  Number of target node: 0
  Number of portals: 3
  Number of session: 0
  Failed session: 0, Last failed initiator name:


There is something in the above that shows what the problem is, but I still couldn't work out what was going on, so I enabled debug ips iscsi msgs:

Here is the output that I got that told me something was up:

SANB# 2013 Mar 28 18:58:51.567443 ips: Session Create (init-name:[iqn.1991-05.com.microsoft:nervmainpc] tgt-name:[] init-alias:[] ISID:[400001370000] indx:[0x02000000] init_ip:[10.0.0.83] 

The above showed me that the IQN was correct and i was coming from the expected IP address..

2013 Mar 28 18:58:51.569116 ips: Session Create Response: Init: node_name iqn.1991-05.com.microsoft:nervmainpc init_name iqn.1991-05.com.microsoft:nervmainpc init-nwwn 21:01:00:0d:ec:2b:3c:42 init-pwwn 21:01:00:0d:ec:2b:3c:42 init-fcid 0 isid 400001370000pgt 12288 num_auth method 2 nodeIndex 2target_name  tgt-nw


2013 Mar 28 18:58:51.576393 ips: Session Destroy node-name:[iqn.1991-05.com.microsoft:nervmainpc] init-name:[iqn.1991-05.com.microsoft:nervmainpc] tgt-name:[]ISID:[400001370000] indx:[0x02000000] failure-code:[1] 

The above showed me that the target list is empty?

 
2013 Mar 28 18:58:51.576929 ips: Session destroy response status 0 for init_name:[iqn.1991-05.com.microsoft:nervmainpc] target_name:[] isid:[400001370000] sent
Suddenly it hit me,

SANB# show iscsi global
iSCSI/iSLB Global information (fabric-wide)
  Authentication: CHAP, NONE
  Initiator idle timeout: 300 seconds
  Dynamic Initiator: iSCSI
  iSLB Distribute: Disabled
  iSLB CFS Session: Does not exist
  Number of load balanced VRRP groups: 0
  Number of load-balanced initiators: 0
iSCSI/iSLB Global information (local to this switch)
  Import FC Target: Disabled  Initiator Plogi timeout: 2 seconds
  Number of target node: 0
  Number of portals: 3
  Number of session: 0
  Failed session: 0, Last failed initiator name:


I wasn't actually importing any targets, I needed this command:


SANB(config)# iscsi import target fc

Now when I connected via the initiator I could see targets:



Here is the relevant debug output:

SANB# 2013 Mar 28 18:49:24.952423 ips: Session Create (init-name:[iqn.1991-05.com.microsoft:nervmainpc] tgt-name:[] init-alias:[] ISID:[400001370000] indx:[0x02000000] init_ip:[10.0.0.83] 
2013 Mar 28 18:49:24.954162 ips: Session Create Response: Init: node_name iqn.1991-05.com.microsoft:nervmainpc init_name iqn.1991-05.com.microsoft:nervmainpc init-nwwn 21:01:00:0d:ec:2b:3c:42 init-pwwn 21:01:00:0d:ec:2b:3c:42 init-fcid 0 isid 400001370000pgt 12288 num_auth method 2 nodeIndex 2target_name  tgt-nw
2013 Mar 28 18:49:24.963807 ips: Session Destroy node-name:[iqn.1991-05.com.microsoft:nervmainpc] init-name:[iqn.1991-05.com.microsoft:nervmainpc] tgt-name:[]ISID:[400001370000] indx:[0x02000000] failure-code:[1]
2013 Mar 28 18:49:24.964295 ips: Session destroy response status 0 for init_name:[iqn.1991-05.com.microsoft:nervmainpc] target_name:[] isid:[400001370000] sent
2013 Mar 28 18:49:28.690124 ips: Session Create (init-name:[iqn.1991-05.com.microsoft:nervmainpc] tgt-name:[iqn.1987-05.com.cisco:05.sanb.01-01.500110a000183180] init-alias:[] ISID:[400001370000] indx:[0x02000000] init_ip:[10.0.0.83] 
2013 Mar 28 18:49:28.691226 ips: Querying NS for target pwwn 50:01:10:a0:00:18:31:80 vsan 50 filter fcid 000f0002
2013 Mar 28 18:49:28.692556 ips: NS target response vsan:[50] fcid:[00cd0000] for target pwwn 50:01:10:a0:00:18:31:80
2013 Mar 28 18:49:28.693656 ips: Session Create Response: Init: node_name iqn.1991-05.com.microsoft:nervmainpc init_name iqn.1991-05.com.microsoft:nervmainpc init-nwwn 21:01:00:0d:ec:2b:3c:42 init-pwwn 21:01:00:0d:ec:2b:3c:42 init-fcid f0002 isid 400001370000pgt 12288 num_auth method 2 nodeIndex 2target_name iqn
2013 Mar 28 18:49:30.935782 ips: Session Create (init-name:[iqn.1991-05.com.microsoft:nervmainpc] tgt-name:[iqn.1987-05.com.cisco:05.sanb.01-01.500110a000183182] init-alias:[] ISID:[400001370000] indx:[0x02000000] init_ip:[10.0.0.83] 
2013 Mar 28 18:49:30.936780 ips: Querying NS for target pwwn 50:01:10:a0:00:18:31:82 vsan 50 filter fcid 000f0002
2013 Mar 28 18:49:30.938088 ips: NS target response vsan:[50] fcid:[000f0000] for target pwwn 50:01:10:a0:00:18:31:82
2013 Mar 28 18:49:30.939453 ips: Session Create Response: Init: node_name iqn.1991-05.com.microsoft:nervmainpc init_name iqn.1991-05.com.microsoft:nervmainpc init-nwwn 21:01:00:0d:ec:2b:3c:42 init-pwwn 21:01:00:0d:ec:2b:3c:42 init-fcid f0002 isid 400001370000pgt 12288 num_auth method 2 nodeIndex 2target_name iqn



In conclusion, even the simplest mistakes are easily identified when your debugging, you should be debugging the minute your having problems in the CCIE DC, you don't have time to waste wondering what is wrong with your config!

I hope this helps someone out there