CCIE DC: Adapter FEX

Hi Guys!

So today I did a very quick lab on Adapter FEX.

What exactly is adapter FEX? What is its relation to VN-LINK and VN-TAG?

VN-LINK is an all encompassing marketing term, so let's ignore that for now, it basically encompasses VN-Tag and a few other bits and bobs.

Adapter-FEX utilizes VN-TAG to implement the idea of Network Interface Virtualization (NIV) which is a way of taking one logical adapter, and splitting it up into multiple adapters!

There could be lots of reasons we might want to do this, especially in this day and age of high levels of virtualization. It is worth noting too that NIV is a standard protocol and I have actually heard of some Dell Blades implementing it too, so this is not a Cisco Only story!

The main reason you should care about it is that it's key to understanding UCS, although when you think of adapter-FEX you typically think of UCS C Series, the reality is that behind the scenes, it's the exact same technology that UCS is using.


Anyway, let's look at how to configure it.

First of all, you need to have a server with a NIV capable card, so that's any Palo series card from Cisco (Which is all the VIC cards and the M81KR's and I believe P81E's)

Got that? Good

Next you need a switch that can perform adapter Fex, that would be a Nexus 7k or a Nexus 5k

Got that? Good, let's configure it

So first of all, enable the technology on the Nexus 5k:

install feature-set virtualization
feature-set virtualization

Next, go to your interface that is connected to the server and enable it for vntag:

int eth1/2
 switchport mode vntag
 no shut
!

 The next step is to configure your actual server to support NIV, now on a UCS C series you do this under the adapter, I can only assume on a UCS it's enabled by default.

It is SUPER interesting to me (that's how sad I am) to see how all the parts that make up UCS are all just things that really at the end of the day Nexus and NX-OS can do, when we look at this config we will see the similarities to UCS B-Series and Fabric Interconnects

Anyway, go to your CIMC management interface on your cisco C-Series, login, go to inventory under the server tab, then go to "Network Adapters", click on "Modify Adapter Properties"


Enable NIV for your adapter and specify a maximum number of VM-FEX interfaces, now this seems a little strange as your configuring adapter-fex, but you still do need to specify a number of channels available (we will talk about that soon ;)








YOU MUST ALWAYS RESTART YOUR SERVER


Sorry for the caps, but at this point there is no getting around it like when you install some software that says "i need to reboot" and you risk it and ignore the reboot command anyway ;), when adding or removing interfaces with adapter-fex I have found you must always restart your server, don't risk it, give it a reboot. You can do it straight from the GUI.

While we wait for the server to reboot, let's chat about some very cool things you can do.

So, basically every vNIC you create in the UCS C-Series manager interface is a "Channel", which is a special way that NIV uses (using vn-tag) to tag traffic as belonging to a particular interface, think of it as a vlan tag for interfaces (so yes, when your doing multiple vlan's down that interface.. it's kind of like vlan inception ;), a tag within a tag within a tag.. like OTV, a header in a header in a header ;))

Anyway, each of these "channels" becomes a vethernet interface, and under said vethernet interface is where you specify the properties of that adapter.

Now, you can manually provision the vethernet interfaces and configure them as you like, OR you can create port-profiles that the palo card can use to automatically! create vethernet interfaces.

Pretty cool

But first, let's look at one manually configured:

So, you already have your interface eth1/2 and it's already setup nicely for switchport mode vntag, next step is to create a vethernet interface:
 

interface Vethernet1
  bind interface Ethernet1/2 channel 1
! 

So the bind command is where I specified that "channel" we talked about, and eth1/2 is our interface configured for vntag.

If you look at the interface at this point, it will be down and out:


switch# show int veth1
Vethernet1 is down (nonParticipating)
    Bound Interface is --
    Hardware is Virtual, address is 0005.73c3.4cc0
    Port mode is access
    Speed is auto-speed

another useful troubleshooting command:

switch# show int virtual sum
Veth       Bound        Channel/  Port        Mac                VM            
Interface  Interface    DV-Port   Profile     Address            Name          
-------------------------------------------------------------------------
Veth1   
Total 1 Veth Interfaces

 

switch# show int virtual status
Interface VIF-index   Bound If      Chan  Vlan  Status     Mode     Vntag
-------------------------------------------------------------------------
Veth1     None        Eth1/2           1     0  Init       Unknown  0    Total 1 Veth Interfaces


It's clear to see we need to create our interface on the actual UCS itself, right now that channel (channel 1) is not seeing any config.. so why should it even come up?

So lets go back to the c-series, back to inventory-network adapters agian and now click on the vNIC tab, then click "Add"
Fill in the details for your interface, being sure to specify a channel number, you will notice lots of VERY interesting settings you can configure here:





Trust Host CoS? Enable PXE Boot? Rate Limit? PCI ORDER???

But wait, There's more!

        
  

Enable Uplink Failover?? If your reading this blog your probably a CCIE DC Candidate, these should all sound pretty familiar, they are all settings you can make in UCS with the B-Series!






Ok, after you have finished admiring this like I was, you need to save the changes and:

REBOOT YOUR SERVER

I can't stress that enough: Every time it didn't work for me when doing this lab, a reboot fixed it, I am almost positive you must reboot each time.

Ok, Let's keep going!

While you wait for it to boot, you might be tempted to issue show int vitual sum again, don't, just be patient and let the server fully boot: i found the interfaces don't come up straight away until the server has gotten quite far into the booting process.

If we wait patiently for the server to boot, eventually the interface will bind:
switch# show int virtual sum
Veth       Bound        Channel/  Port        Mac                VM            
Interface  Interface    DV-Port   Profile     Address            Name          
-------------------------------------------------------------------------
 Veth1      Eth1/2       1                  

Success! Our interface is up and bound!


 let's look at what we can do next.
So in UCS, if you have looked at the config,  all your vethernet interfaces are actually port-profiles, port-profiles can be damn useful for grouping together ports that share common attributes.

We are going to use one for Adapter FEX



vlan 10,20
 
port-profile type vethernet ccierants1
  switchport access vlan 10
  state enabled
!
port-profile type vethernet test
 switchport access vlan 20
  state enabled
!

Let's go back to our adapter properties, then back to our vNIC, you could either look at the existing or create a new one, go to the vNIC and scroll down until you see port-profiles

How cool is that! It downloaded our port profiles for us, we can now select one and all the settings will inherit from the switch! Very cool


Select the port profile and click save changes, we need to go back to the switch and give permission for vethernet interfaces to be created:


switch# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
switch(config)# vethernet auto-create
switch(config)#

At this point we have one more step before this is ready, can you guess what it is ;), that's right

REBOOT YOUR SERVER


Now our interface is up ready to go:

switch# show int virtual sum
Veth       Bound        Channel/  Port        Mac                VM            
Interface  Interface    DV-Port   Profile     Address            Name          
-------------------------------------------------------------------------
Veth32769  Eth1/2       1         ccierants1
Total 1 Veth Interfaces


Pretty cool right?