CCIE DC: The confusion of firmware in UCS (Hopefully demystified)

Hi Guys

So this blog post is going to focus on UCS 2.0 and the work on firmware, I was watching some vendor training videos on this and to be honest they where somewhat lacking and really glossed over the details, so I wanted to really dive in.

(For those of you reading this and hoping to learn about firmware management for your production UCS, please note this is really focused on UCS 2.0, in UCS 2.1 apparently there is a nice firmware update wizard which makes this whole process not as complicated)

So first of all, it's worth noting: almost every component in UCS has dual-firmware flash, the blades do, the adapters do, the IOM modules do, and the fabric interconnects do.

When you go to Equipment - Firmware Management, and you see "download" to UCS, what you are doing here is actually copying the firmware to the UCS so that it can place it in to the appropriate file locations. When you "update" the firmware, what you are doing is telling UCS to copy the new firmware into the second flash, the backup firmware flash.


THIS IS PERFECTLY SAFE TO DO "live", you won't cause yourself any issues at all.
It also saves you tons of time when you actually need to change the firmware as all your firmware is already updated ready to go. Great!

Activation actually tells the device to change which flash is the "Active" flash, kind of like when you switch partitions on CUCM, when you tell it "set startup version", the device your activating is not reset and instead will run that new firmware when you reset.

Cisco recommend that you update all devices from the inside out, so starting with the adapter, moving up to the blade components, then to the IOM, then to the fabric interconnect, but there is actually a set of steps you need to take if you want this to work seamlessly.



First thing is you want to update the Adapters, you can do this on each server by first either migrating all the guest VM's off it (if it's a vmware server) or migrating the service profile (which obviously still has a short downtime as you migrate the service profile and reboot the server. You could also upgrade one adapter  at a time.

Next, the CIMC can be updated, this is a very safe operation to do as NO INLINE TRAFFIC will be affected either for the vHBA or the vETH, you won't lose connectivity to your server, you will lose KVM access, IPMI access and Serial Over Lan however, but this is a relatively safe operation.


The next step is to activate the IOM Modules, HOWEVER remember an IOM module is just a FEX at it's heart, and where does a FEX get it's firmware from? It gets it from the host N5k, well in this case the IOM gets its firmware from the Fabric Interconnect, so if you for example, updated the IOM firmware, activated it, then restarted it, guess what, it's going to boot, say to itself based on what the FI has as it's firmware "Well dang, I have the wrong firmware", download the "old" firmware from the FI and then reboot.

Woops, that wasn't exactly helpful for you was it?
So, you just want to say on the IOM, "set startup version", when the FI is reset, the IOM module will launch the correct "new version"

Upgrading the FI's is relatively straight forward as long as you do one at a time (which should be obvious), it's still recommended to do this in a scheduled maintenance window and I always would too, but depending on your failover setup, if you have done it properly, it should stay active.

NOW HERE COMES THE CRITICAL PART

 Some people, and the video i watched seemed to imply your done now, that's it, firmware updated.. Wrong, there are now specific blade firmwares that CANNOT be updated using this method, more than likely in my opinion because for the below items, they do NOT support dual firmware.

These components are:
BIOS
RAID Controller
HBA
HBA Option ROM

The BIOS image is a strange one, you might have seen "Recover corrupted bios" under your blades on the main tab of your blade, but this is NOT a valid method of upgrading your BIOS and really should be used for if you ever hose your BIOS.


To create a policy for the actual blade itself, go to:

Servers Tab
Policies - Host Firmware Packages

From here you can select the individual firmware packages that the operating system will see, these are the critical aspects from the operating system point of view, the other firmware that you updated from the Equipment tab are really transparent to the OS, the OS really doesnt deal with that firmware, but the firmware you specify here he sure does.


Once you have created the host firmware package policy, you actually apply this to your service profile.

Easy Guys! I hope this helps someone out there