CCIE DC: Understanding Failover and network config in UCS

Hi Guys

In this blog post I am going to try and explain the network configuration in UCS. Many people have difficulty understanding the failover and how it should be configured. I recently explained the concepts around it to a customer and I think that it made a lot of sense to them so i wanted to share it here using the same methodology.

I want to cover off both North-Bound (I.e. the uplinks back towards the network) and South-bound (I.e. towards the servers) in this article.

South-Bound

Since I am from Australia let's start with South Bound. 

First, picture in your mind a switch chassis, with a single line card, and a server attached to this linecard with a single nic



Pretty simple to picture so far hey? OK, it's obvious that this setup has quite a few single points of failure, so let's expand the picture, introduce another entire switch, and another NIC, each acting independently of the other (so no VSS or vPC or anything complicated like that :))

Got it? Great!

Now we have redundancy, and no single point of failure, if either of the switches or NIC fails you have full redundancy.

Congratulatons! You have now accurately pictured the way Cisco UCS does south-bound connectivity.

In UCS, Each Fabric Interconnect is an entirely independent switch, and the FEX is just a remote line card (this is litreally what happens in UCS, if you look at the config on the FI it is just a Nexus 5k with a 2k remote line card that happens to sit in the chassis)

With me so far? Easy right!

OK so let's take this a step further, if your server only has one NIC, what could you do? well in  real life not much, but with Cisco UCS each vNIC is actually just a virtual instance, the server itself has a mezzaine card, but how many NIC's the server has is actually entirely up to you, it can have up to 64 if you really wanted! but they are virtual.

So when would you ever define just one NIC? maybe your operating system is not intelligent and can't provide redundancy across two NIC's, in this case you would use UCS's built in failover! the built in failover means that if something happens to FI A, FEX A, or FI B or FEX B, Cisco UCS can handle that failover!

If your OS supports it, you could rely on the OS to provide the failover, in ESXi you would configure the vSWITCH active and standby adapters and be all good.

Now let's say however that you had a vSWITCH and configured up two vNIC's, in the back end, on Cisco UCS, one of these vNIC's is going to Fabric A, the other to Fabric B, you go to your vswitch and you configure etherchannel across these NIC's

Your going to have a bad time :(, If you think about it, your trying to etherchannel across two totally separate switches, you will have very strange problems such as Ping's dropping in and out and other issues. It will be quite hard to track down! So remember, you can't etherchannel across two NIC's if those NIC's actually go back towards Cisco UCS on two separate Fabrics! Because each fabric is a separate switch!

My question has always been: Why even have multiple NIC's at all? Just have one going to each of your servers and let UCS provide the failover, after all UCS is going to notice a failure sooner!

But the problem with this approach is that VMWARE will complain if it does not have two NIC's attached, you will receive a message in the console complaining of single NIC connectivity. Therefore many people like to ensure there are two NIC's.

For your information, when a fault occurs with either a FEX or the FI on a fabric, the NIC's will physically disconnect in ESXi so you can always use physical status as your failover detection in ESX :).

Now that you understand what is actually happening in the background, the possibilities for your ESXi Switch configuration are probably starting to make sense, you could have one giant vSWITCH for everything with two Adapters, one active one standby, you could have one giant switch and have both adapters active but use "load balance based on originating virtual port ID" which will send some traffic from one host down one link and some traffic from the other host down another. (Incidentally, i wouldn't get too caught up on "using all the available bandwidth" because the fact is that in the backend, the traffic all goes to one mezzaine card anyway! which has 10 gig or 20 gig or 40 gig of throughput, if you want to control bandwidth you should be using the QoS settings on the NIC in UCS but that is a conversation for another blog)

You could even split traffic up into two seperate vSWITCHES, one for management one for data and have active/standby adapters in each of these. But the important thing to think about especially when you start running into weird problems with pinging etc is to remind yourself of what the topology actually looks like. All this virtualization etc can make the topology more difficult to picture, but if you understand the topology you will quickly work out what is going wrong.



North-Bound 


Now that you understand south bound, take yourself back to our picture in our mind of two separate switches with line cards and a server dual-attached to them. Ask yourself this, for the upstream switches that you will be attaching your imaginary switches to, would you dual attach them? Of course you would, if you could vPC or VSS etherchannel them to two seperate switches, would you? Of Course you would!

It is no different in the UCS World, your Fabric Interconnects A and B should be dual attached to your upstream switches wherever possible, be that by using vPC or etherchannel.


A big no-no is having each Fabric Interconnect uplinked to a single Nexus 7k each, but then those Nexus 7k's being in a vPC, because this means that whenever a host on fabric A needs to talk to a host on fabric B it goes over your internal link that your Nexus 7k's use (in this case it would be the vPC Peer link, in another example like VSS it would be your VSS Link)

The Take Aways

If your the type who looks for too long didn't read, this is your section :)

• The best possible way to think of each Fabric Interconnect is to think of them as a seperate switch, with a remote line card (the FEX), because in reality, that is actually what they are, the Fabric Interconnect is just a Nexus 5k and the FEX is just a Nexus 2k.
• The fabric Interconnects are NOT vPC'd together, they are two totally independent switches
• Thus, when you create vNIC's in Cisco UCS, and assign them to either fabric A or Fabric B, you need to picture in your mind "I am setting uplinks up to TWO diffirent switches" and all of a sudden your failover and etherchannel etc configuration will make sense
• Speaking of failover, your choices are to rely on the operating system or Cisco UCS to do the failover for you, but never do both
• Always uplink your fabric interconnect's to BOTH upstream switches and use etherchannel whenever possible. (So FI A should attach to Core Switch A and B and its link's should be in an etherchannel!)

I hope this helps someone out there

Recommended HTTP Configuration for ACE Load Balancing

Real quick one guys, i recommend the following ACE Load balancing config for HTTP Load Balanced sites:

parameter-map type http HTTP_PARAMETER_MAP

  case-insensitive

  persistence-rebalance

  set header-maxparse-length 65535

  set content-maxparse-length 65535

  length-exceed continue

  parsing non-strict

Occasional confusing issues on Nexus 5000 Layer 3

Hi Guys

This is a short post as it is really more for my own benefit, i had a hell of a time getting a Nexus 5000 to work correctly due to some misunderstanding on my part with Nexus Licensing.

So, Nexus 5000 with a layer 3 daughtercard is ENTITLED automatically to LAN Base.

Now let's say you installed LAN-Enterprise, which is meant to come with all the features of LAN Base, they are actually seperate licenses and if you have one but not the other you will have very strange issues, I had an issue where VRRP would not recognise each other as neighbors even though it let me configure it! Unlike a 3750 for example when you have to say "ip routing" in order to have the device perform layer 3 routing, the nexus does not have any similiar command and infact you will be able to configure all the L3 features you want, they just won't work!

So if you have weird problems where the nexus 5000 won't route traffic even though you have the layer 3 daughtercard and entries in your route table, be sure to check you have the LAN Base image, it's free with the layer 3 daughtercard you are entitled to it :).  Check below to see where to get it from.


Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
FC_FEATURES_PKG               No    -   Unused             -
LAN_BASE_SERVICES_PKG         Yes   -   In use Never       -
LAN_ENTERPRISE_SERVICES_PKG   Yes   -   Unused Never       -
--------------------------------------------------------------------------------
!

If you don't have the LAN BASE license listed above you will be hosed, to get a copy just go to www.cisco.com/go/licensing, click on get demo, nexus 5000 and then select LAN Base license, you will see it's a permanent license you can install.

Understanding Fiber Link Loss budgets

Hi Guys!

Thanks to a co-worker of mine I now have a pretty good understanding of fiber link loss budgets and how to calculate them (Or do I? Post below if you think i have made an error in some calculations or understanding, I love hearing from you guys!)

Let's briefly chat about what we are trying to calculate here and why it is important.


As you may know, fiber optic cable works by using light from a transmit and receive pair on a optic cable (although in some topologies this is done on a single strand of fibre using a TDM method but we will leave that out of this discussion)

As light travels over longer distances it dissipates and becomes weaker, sometimes it is so "dull" that the photosensitive receiver at the other end cannot see the signal, conversely sometimes the light source is so bright the receiver is overloaded, like staring into a bright light or the sun.

So, when choosing fiber optics and deciding if you need an attenuator (which is like sun glasses for the SFP) or a power booster, you need to take into account the distance of the fibre, and from there you can calculate the link loss, then by using the cisco data sheets you can calculate the exact SFP you will need.

Your best bet when performing this is to speak to your fiber optics cable installer and ask for the OTDR report, this report contains information from the installer as to his measured span loss, the loss is normally listed in DB.

If you don't have this report, you can only really guess based on the average loss per km over typically installed fiber, note that diffirent installers produce varying quality when installing fibre so this is not the preferred method, seek out an accurate OTDR report if you can! The average loss per KM for a fibre run is between 0.2 to 0.5 per KM, although higher has been reported! Splices on the fibre also introduce a certain amount of loss between 0.2 to 0.5, and finally patch panels can also add similiar levels of loss, hence your always better off with an OTDR report!

Let's talk a little more about the OTDR report, first of all you should check to make sure that the wavelength that the installer used when he ran the test is the same wavelength that your SFP is going to use, lower wavelength has less power therefore it travels shorter distances.

Once you have this info, let's perform our calculations!

So first of all, you need to know the maximum transmit and receive power of your SFP:

Let's take a typical example which is the ER module: (SFP-10GB-ER)
(all values are in the datasheet) http://www.cisco.com/en/US/prod/collateral/modules/ps5455/data_sheet_c78-455693.html


Transmit Power

Maximum: 4.0
Minimum: -4.7

Receive power:
Maximum: -1
Minimum: -15.8


If you compare these values to a shorter range such as the LR: (SFP-10GB-LR)

Transmit Power

Maximum:0.5
Minimum:-8.2

Receive power:
Maximum:0.5
Minimum:-14.4

(all measurements in dBm)
 You can see from the above that the ER has signifcantly more transmit power, interestingly the receive power on the LR as a maximum is quite high, I suspect this might be to prevent people accidentally burning out short-range LR modules, I would be keen to hear more from someone on this.


So how do we calculate if our SFP is going to be suitable? Let's assume you have an OTDR report and the report says that the loss is going to be 5.6 dBm, do the following:

Maximum transmit power minus Loss in OTDR Report = X
Minimum Transmit power minus Loss in OTDR Report = Y

As long as X and Y are below the maximum receive power but ABOVE the minimum receive power, you will be fine, lets work this out:

0.5 - 5.6 = -5.1
-8.2 - 5.6 = 13.8

As you can see we are safely in the margins.

You can view at any time on a SFP with DOM support the current details of its transmit and receive power:


show int eth1/50 transceiver details
Ethernet1/50
    transceiver is present
    type is 10Gbase-LR
    name is CISCO-FINISAR

           SFP Detail Diagnostics Information (internal calibration)
  ----------------------------------------------------------------------------
                Current              Alarms                  Warnings
                Measurement     High        Low         High          Low
  ----------------------------------------------------------------------------
  Temperature   33.25 C        75.00 C     -5.00 C     70.00 C        0.00 C
  Voltage        3.34 V         3.63 V      2.97 V      3.46 V        3.13 V
  Current       32.75 mA       70.00 mA     1.00 mA    68.00 mA       2.00 mA
  Tx Power        0.46 dBm       3.49 dBm  -12.21 dBm    0.49 dBm     -8.21 dBm
  Rx Power       -1.24 dBm       3.49 dBm  -18.53 dBm    0.49 dBm    -14.43 dBm  ----------------------------------------------------------------------------
  Note: ++  high-alarm; +  high-warning; --  low-alarm; -  low-warning
As you can see the warning values match up nicely with the information available in the data sheet, curiously the Alarms are actually quite high values, but when it comes to optics you probably want to stay well within the warning levels and not push towards alarm levels!

ISDN Layer 1 Status

Hi Guys!

If your anything like me, you thought that an ISDN output like below:

#show isdn status
Global ISDN Switchtype = primary-net5
ISDN Serial0/0/1:15 interface
        dsl 0, interface ISDN Switchtype = primary-net5
    Layer 1 Status:
        DEACTIVATED


Meant that there was some sort of cabling problem, I always assumed Layer 1 deactivated means there is a cabling problem and that is that, but alas no! It can mean a few things, Check out the output below:



#show controller e1 0/0/1
E1 0/0/1 is down.
  Applique type is Channelized E1 - balanced
  Description: ### E1 Outside Line ###
  Transmitter is sending TS16 LOMF Alarm.
  Receiver has loss of multi-frame in TS16.

The above output shows that you can see a little bit of info about what Alarms the E1 Can see, this is another way of accomplishing more troubleshooting, From the output above we can see it is upset with the framing in some way, so let's change the framing:



controller e1 0/0/1
   framing no-crc4

Now let's check the status:
 

#show isdn status
Global ISDN Switchtype = primary-net5
ISDN Serial0/0/1:15 interface
        dsl 0, interface ISDN Switchtype = primary-net5
    Layer 1 Status:
        ACTIVE
    Layer 2 Status:
        TEI = 0, Ces = 1, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED


Hooray!!! Another way to effectively troubleshoot E1 Lines.