Boot from SAN iSCSI with Cisco UCS 2.0

Update:

Here are a couple of tips for all of you, if you see the error message about invalid iSCSI Configurations when configuring this and trying to apply as a service profile, first of all to help troubleshoot try removing the iSCSI boot parameters so that you just have the actual iSCSI NIC's but no iSCSI boot policy, that way you know if the error is related to your iSCSI NIC or your iSCSI boot policy. 


One particular error for me was that I had defined a MAC address pool for my iSCSI adapter the actual vNIC itself under iSCSI NIC's, i had actually set a MAC pool, for M81KR Adapters you should set this to derived or not select a pool at all.


 

 Hi Guys!

Hot on the heels of my previous post about Port Channels with UCS between the IOM's and the Fabric Interconnects (a new feature in UCS 2.0) comes another blog post about another great new feature in UCS 2.0.

I wrote this article as although there is a few out articles already out there talking about boot from SAN iSCSI, it is very new and thus troubleshooting it is a bit more difficult, I also want to show you how to make sure it uses Jumbo Frames :)

Sections of Document

Boot from SAN iSCSI

Troubleshooting boot from SAN iSCSI

Boot from SAN iSCSI

In this blog post, I will explain how to configure boot from SAN iSCSI and a great way to TEST it yourself without having to involve any SAN guys until your certain your end is all working great :), with some very handy FREE software

Finally, I will take you through some great ways to troubleshoot using some very cool and useful commands available under your Mezzanine Adapter (A CLI in my Mezzanine Adapter? what a cool concept! and extremely useful!)

Let's get started

First of all here are the enviromentals used here just so we can make sure we are all on the same page :)

We are using UCS 2.0(X), The Cisco M81KR Mezzanine Cards (Other cards SHOULD be fine but you may need to adjust your iSCSI adapter policies) and some B2XX series blades

So, first step is to login to UCS, I am going to assume you already know how to create a service profile, so I have just created one and we will be modifying that to suit boot from iSCSI :)

The first step is to go to the LAN tab and create a VLAN for our iSCSI traffic, I try (unless I can't avoid it) to not route iSCSI traffic, as this can stuff around with the MTU etc. But ofcourse that doesn't always fit with every situation

Go to the LAN TAB -> LAN Cloud -> VLAN's






Create a new VLAN with the plus button on the right hand side, give it a suitable name
and more than likely, you are just going to set it as common/global

give it an ID too :)





Now we need to create some QOS policies for our vNIC's

Under the same LAN tab, under LAN cloud again, go to "QoS System Class"
Pick a class you are not using for anything else, I picked GOld, Set the class to enabled,
 and change the MTU to 9000, I leave packet drop for this class turned ON, I know that some people think this is not a good idea, but my personal feeling is that iSCSI unlike fibre channel, has mechanisms in place to gracefully deal with packet drop and that packet drop should be reserved for traffic that REALY needs it, but anyway

Here is a screenshot of the gold class configured for this




Next, we need to assign this class of traffic to a QOS policy. Go to Policies - Root (or your own sub-organization if your using them (and if your not, why not?) and then click on QoS Policies, create a new QoS Policy from here, Change the priority to whichever class of traffic you chose in the previous step.



Next, let's create some vNIC templates :)

In the LAN tab go to vNIC Templates, we will be adding two new vNIC templates.

Set the name as you like, for the fabric ID, choose fabric A for this, and the next vNIC we create we will be choosing fabric B. I don't personally enable failover, I like to treat it like fibre channel where you have two separate storage networks, but the ticking/unticking of this box will depend on your own iSCSI topology :)
 (Got an opinion as to why one or the other is better? Leave a comment!)

I personally set the template type to updating, so that if I have made a mistake, all my service profiles are updated along with it.

 Select your iSCSI VLAN you created previously, and set that as the native VLAN

Change the MTU to 9000

For the MAC pool, select a MAC pool that you have created previously (or create a whole new one if you prefer :))

The QOS policy should be the same QoS policy we created in the previous step.






Now it's time to associate these two vNIC's with your service template, the next few steps may differ if your setting up a service profile from scratch, i am going to for the sake of brevity assume your modifying an existing service profile.

Go to the server tab and find your service profile, click on vNIC's and click Add

Name your  iSCSI adapter and click "use LAN connectivity template, then select your vNIC template you created previously, repeat this for your two vNIC's using the separate fabric failovers




 Now, click on your service profile and click on iSCSI vNIC's, then click add, give your iSCSI vNIC a name (in this case, i chose iSCSIHBA1, and for overlay vNIC choose the vNIC we assigned to the service profile previously, for the VLAN select the iSCSI VLAN we created previously.




Make SURE you don't specify a MAC Pool but instead choose none, I had a major problem where I set a MAC pool, it said that the configuration was wrong and configuring this became quite difficult, I was told that the iSCSI configuration was incomplete (at least that is what the Service Profile indicated), this might be an issue with 2.0.1(w), but if your experiencing this issue, that is likely to be your problem.

Ok! We are getting close now!

So just to review, you should now have a System Class for QOS, a QOS Profile, some vNIC templates, two vNIC's created based off these templates and assigned to your service profile and finally two iSCSI NIC's now showing under your Service Profile

*Phew*


OK, the last step is to modify the boot parameters so that it will boot from iSCSI, click on the top branch of your service profile tree (the service profile itself) and select Boot Order.

Click "Modify Boot Policy", select "specific boot policy" (You could also go and create a boot policy for this under your organization)

First, add a CD-ROM drive as you will need to be able to actually install your OS so I am assuming you will use a KVM with virtual media to do this :)
Next, click on your iSCSI interfaces you created previously, and assign them into the boot policy, then click OK











Next, we need to define the boot Parameters


You will be taken back to the boot order tab when you click OK. Click on "Set Boot Parameters" under the iSCSI vNIC's on the left hand side, we will need to set Parameters for both.

The first step is that you must choose an initiator name, I chose to use the following format:

iqn.1992-08.cisco.com:2500
and
iqn.1992-08.cisco.com:2501 

for each of my vNIC's, they must be separate for each of your vNIC's

 For the initiator IP address policy, this is where you actually set the IP address of the iSCSI adapter itself, so this should be something in your iSCSI VLAN and it should be able to reach the target






Next, at the bottom of this screen is where we specify the target, click Add and enter in the name and IP address of your target




Repeat these steps using different initiator names and  IP Addressing for the other iSCSI vNIC boot parameters.

Now, _if_ your confident that your iSCSI target is correct and setup and ready to go (and supports boot from SAN) at this point your finished! You can go ahead and boot the system, install your software to the remote LUN over iscsi and you should be good to go.

Let's assume for a minute that its not quite that simple ;) and that you come across issues. Let's start talking about troubleshooting!

Troubleshooting Boot from SAN iSCSI

So of course, before you can even begin to install the operating system, you actually have to be able to see the LUN. Let's talk about how to make sure that is happening.

First of all, when you boot your freshly configured Service Profile, you should notice that the vNIC actually comes up with a message during boot saying if  it was able to see the target, if your all good, you will see it show the LUN available via your target, if your not all good it will show something such as Initiator Error 1 (which generally means it couldn't find the target) or initiator error 4

But sometimes those messages scroll by so fast, and it takes a while for the server to reboot! So sometimes you just want to be able to find out if it worked some other way, and hell, get a bit more information too!

So, inspired by another Bloggers post I found, where he talks about how to see what LUN's are available over a HBA before and after the server has booted

 
I wondered if there was a way to do the same thing in iSCSI? there is!


First, login to the UCS CLI, then connect to the adapter of the server that is running the Service Profile with the boot from iSCSI (Where in my example, 1 is the chassis ID, 1 is the server, 1 is the mezzaine adapter number)

UCSHOSTNAME#connect adapter 1/1/1
UCSHOSTANME#(adapter)connect
UCSHOSTNAME(adapter)#attach-mcp

adapter 1/1/1 (mcp):20# iscsi_get_config

vnic iSCSI Configuration:

----------------------------

 

vnic_id: 6

          link_state: Up

       Initiator Cfg:

     initiator_state: ISCSI_INITIATOR_READY

initiator_error_code: ISCSI_BOOT_NIC_NO_ERROR

                vlan: 0

         dhcp status: false

                 IQN: iqn.1992-08.com.cisco:2500

             IP Addr: 192.168.227.10

         Subnet Mask: 255.255.255.0

             Gateway: 192.168.227.1

          Target Cfg:

          Target Idx: 0

               State: ISCSI_TARGET_READY

          Prev State: ISCSI_TARGET_DISABLED

        Target Error: ISCSI_TARGET_NO_ERROR

                 IQN: iqn.2008-08.com.starwindsoftware:192.168.227.10-test

             IP Addr:192.168.227.10

                Port: 3260

            Boot Lun: 0

          Ping Stats: Success (9.982ms)

        Session Info:

          session_id: 0

         host_number: 0

          bus_number: 0

           target_id: 0

You can tell from the very helpful output above that there was no issue connected to the iSCSI target!, this will also give you useful information about if the target did fail for some reason, and why, you can even run iscsi_ping to make sure that you can actually reach the target via network.

So all in all a very cool way to do troubleshooting.

And Another Thing...

So maybe you don't have access to the SAN storage yet, and your trying to test boot from SAN iscsi, or maybe your trying to ensure the problem is not on the UCS side but possibly with the SAN, how can you quickly and easily run up a SAN? I mean they cost tons of money!!


Enter the Starwind Software iSCSI software SAN, a _FREE_ version is available on there website, and runs on ANY windows platform (not just windows server platforms) and allows you to quickly and very easily create an iSCSI SAN, you don't even need Raw Disks!!! you can just tell it to create a file on the computer and treat that as a volume/LUN to be exported via iSCSI (Very cool!)


When you download it and install it, just so you know the default username and password (which is not documented very well) is root and starwind


As you can see, it was super easy for me to create a iSCSI target, i just clicked next a few times and away i went, and best of all this software is FREE (some features are paid), believe me when I say i am not being paid to push this product (no one pays me anything :p) I am just so impressed that someone could offer this software for free when it works so incredibly well.


It is my sincere hope that armed with this knowledge you will be able to go out there and prove UCS iSCSI boot from SAN works perfectly for Cisco UCS (I wonder when Junipers Server products are coming? ;))

I hope this helps someone out there!