CCIE DC: Nexus 1000V troubleshooting

Hi Guys!!

If your anything like me, one of the most nerve-racking things when installing the Nexus 1000V is the big step when you move the uplinks of your ESX host to the Nexus 1000V. this can be quite daunting, but is a perfectly safe operation that you can always back out of. In this I will take you through the steps of troubleshooting and resolving issues with this connection.


Incidentally, one of the main hints in this article, how to add a system-VLAN using the command line on the ESX Server was pointed out by one of the many very smart people on our CCIE DC facebook group, I strongly recommend if your serious about CCIE DC that you join this group, we have lot's of really smart people contributing some great ideas to this group.

So full credit to my fellow members of this group!


Here are some general recommendations
 - Do one uplink at a time if you can, keeping your host available indirectly if you get absolutely stuck

- Be sure to have organised KVM or ILO access to the host so that if you need to get onto the host in a hurry you can

- Ensure you have enabled SSH and direct console access to the host before you move the uplinks so you can login to troubleshoot

- Of course, put the host in maintenance mode before you perform this operation :).

OK so let's talk about the System VLAN.


The System VLAN command you define under your port profile is the single most important command you can do to ensure a safe change of your uplinks: The VLAN's you specify as System VLAN's bypass the distributed switch and are allowed to communicate with no restrictions. Your System VLAN should match your VMKernel Port.

Here is an example of one done correctly:

port-profile type ethernet n1kv-eth-2
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 1
  switchport trunk native vlan 1
  channel-group auto mode on sub-group cdp
  no shutdown
  system vlan 1  state enabled
 In this example, I have specified that VLAN 1 is my system VLAN. this is the VLAN that my VMKernel Management Port uses:

port-profile type vethernet n1kv-veth-vlan-1-l3
  vmware port-group
  port-binding static auto
  switchport mode access
  switchport access vlan 1
  no shutdown
  system vlan 1
  max-ports 256
  min-ports 16
  state enabled




You need to specify the system VLAN both on the uplink and on the vethernet that you will be assigning to the host VMKernel itself.

OK, let's look at an example where I was naughty and didn't assign it correctly:

Here is my ethernet port that doesn't have a system-VLAN assigned

port-profile type ethernet n1kv-eth-2-no-sysvlan
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 1
  switchport trunk native vlan 1
  channel-group auto mode on mac-pinning
  no shutdown
  state enabled




I assign this to my host and oh dear, i can no longer access the host:




To be honest, if you have done this and you've forgotten to specify a system VLAN, your pretty much hosed and will need to restore to standard Switch, let's take a look why:





In the above example, you can see that the admin state for the physical interface (vmnic0) is down, this will occur because you have not specified a system VLAN for one of the uplinks, you will notice that vmk0 is showing as up up because this port HAS been assigned (correctly) a system VLAN. Ports will remain down until they can talk to the VSM, and they can't talk to the VSM if they don't have a System VLAN....


So in this case your only option is to restore the standard vSWITCH.

Let's take a look at what happens if we specify an incorrect System VLAN on both our vethernet and ethernet profile, and we also configure completely the wrong trunk:

port-profile type ethernet n1kv-eth-2-wrong-sysvlan
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 1-2
  switchport trunk native vlan 2
  no shutdown
  system vlan 2
  state enabled
 
port-profile type vethernet n1kv-veth-vlan-1-l3-wrongsysvlan
  capability l3control
  vmware port-group
  port-binding static auto
  switchport mode access
  switchport access vlan 2
  no shutdown
  system vlan 2
  max-ports 256
  min-ports 16
  state enabled
!


Let's look at our helpful show port output now:



So the first step is to restore the trunk so that the correct VLAN is the native VLAN

 

We are now in the correct VLAN for our trunk, a few more steps remain:

 

Let's see what we can see under show bd now, which is a very useful command for determining which interfaces are using the VLAN:

This looks pretty promising: we can even see the multicast group, let's check to see if we now have MAC address table entries:





Success! So finally we should have show module....

FakeNexus1000V# show module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
3    248    Virtual Ethernet Module           NA                  ok
4    248    Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw     
---  ------------------  ------------------------------------------------ 
1    4.2(1)SV2(1.1a)     0.0                                             
3    4.2(1)SV2(1.1a)     VMware ESXi 5.0.0 Releasebuild-469512 (3.0)     
4    4.2(1)SV2(1.1a)     VMware ESXi 5.0.0 Releasebuild-469512 (3.0)     

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA
3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA
4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    172.21.1.11      NA                                    NA
3    172.21.1.103     564d22bf-1f05-4824-33e3-d3da94c45c17  NA4    172.21.1.105     564d0f85-3b7e-313e-d774-1124ea22cf10  172.21.1.105
We do! Great Success!


So, as long as your uplink (in our case, LTL 17), your VMKernel (LTL 49), Control (Always LTL 10) and packet (always LTL 12) have the system VLAN specified (determined using show bd you will be able to recover from the incorrect System VLAN.


Two other great troubleshooting commands are vemcmd show port-old:


and vemcmd show card, which can be used to determine if you have an issue with duplicate VSM's





I hope this helps someone out there!

CCIE DC Study: Nexus 1000V

Hi Guys

This post is going to be about the nexus 1000V in order to help those out there studying for the CCIE DC. I am not going to go through in this blog post how to install the Nexus 1000V as this is covered in great detail elsewhere. Instead I am going to talk about why you might want to use the Nexus 1000V and some of the features available to you.

First of all, before we go any further did you know that the Nexus 1000V is now COMPLETELY FREE? 

That's right, Nexus 1000 now comes in two editions, "Essential edition" which is completely free to use and Advanced edition. You can go to cisco.com right now and download your very own copy of Nexus 1000V.

Now that we have that out the way, let's look at some of the Security features the Nexus 1000V has, later we will look at QoS and (if we get time) VXLAN


DHCP Snooping
This feature might be familiar to you as it is available on existing physical switches from Cisco right now, what you need to ask yourself though is as VDI Deployments increase, how do I protect my virtual infrastructure? If your doing a "cloud" deployment and your intending to have lots of machines from lots of diffirent locations running VDI the chances of one of those users being a nefarious hacker increases, so the necessity to protect your infrastructure becomes more vital. DHCP Snooping allows you to protect against some common Man-in-The-Middle attacks as well as a few sophisticated attacks we will chat about.

Let's briefly chat about the configuration I have. I have two windows Servers configured, one is acting my as DHCP Client and the other as my DHCP Server, both are connected on VLAN 50 (172.21.5.0/24) on my Nexus 1000v.

The first thing to do is enable the dhcp feature:


DCNexus1000V(config)# Feature dhcp

You may receive an error when you attempt to do this: this does require the advanced edition of the Nexus 1000V, you can enable the advanced edition as a trial for 60 days with:


DCNexus1000V(config)# svs switch edition advanced

Once this is done and you have now entered the feature DHCP command, enable ip dhcp snooping for the appropriate VLAN:

DCNexus1000V(config)# ip dhcp snooping vlan 50




Let's look at the configuration so far:

 DCNexus1000V# show ip dhcp snooping
Switch DHCP snooping is enabled
DHCP snooping is configured on the following VLANs:
50
DHCP snooping is operational on the following VLANs:
none
Insertion of Option 82 is disabled
Verification of MAC address is enabled
DHCP snooping trust is configured on the following interfaces:
Interface             Trusted        Pkt Limit
------------          -------        ---------
Vethernet1            No              Unlimited
Vethernet2            No              Unlimited
Vethernet3            No              Unlimited
Vethernet4            No              Unlimited
Vethernet5            No              Unlimited
Vethernet6            No              Unlimited
Vethernet7            No              Unlimited


As you can see from the above output DHCP snooping is still not active, so let's go ahead and enable the entire DHCP feature itself with:

DCNexus1000V(config)# ip dhcp snooping 

Now DHCP snooping will be enabled for VLAN 50.

DCNexus1000V# show ip dhcp snooping
Switch DHCP snooping is disabled
DHCP snooping is configured on the following VLANs:
50
DHCP snooping is operational on the following VLANs:
none
Insertion of Option 82 is disabled
Verification of MAC address is enabled
DHCP snooping trust is configured on the following interfaces:
Interface             Trusted        Pkt Limit
------------          -------        ---------
Vethernet1            No              Unlimited
Vethernet2            No             Unlimited
Vethernet3            No              Unlimited
Vethernet4            No              Unlimited
Vethernet5            No              Unlimited
Vethernet6            No              Unlimited
Vethernet7            No              Unlimited

Let's take a look at the default behavior of DHCP Snooping, as you can see from the above all ports are in the untrusted state, which means that if i try and get a DHCP address on my Windows Server...

Windows 2008 DHCP Client:
ipconfig /release

ipconfig /renew


The request just hangs, i can see that the DHCP responses are being blocked:

 DCNexus1000V# show ip dhcp snooping statistics
Packets processed 23
Packets forwarded 22
Total packets dropped 1
Packets dropped from untrusted ports 0
Packets dropped due to MAC address check failure 0
Packets dropped due to Option 82 insertion failure 0
Packets dropped due to o/p intf unknown 0
Packets dropped which were unknown 0
Packets dropped due to service dhcp not enabled 0
Packets dropped due to no binding entry 0
Packets dropped due to interface error/no interface 0
Packets dropped due to max hops exceeded 0

Therefore I must configure a port as trusted with a trusted port-profile:

 port-profile type vethernet DHCP_TRUSTED
  vmware port-group
  switchport mode access
  switchport access vlan 50
  no shutdown
  state enabled
  ip dhcp snooping trust

I then assign this port profile to my DHCP Server and try renewing the address again

DCNexus1000V# show ip dhcp snooping
Switch DHCP snooping is enabled
DHCP snooping is configured on the following VLANs:
50
DHCP snooping is operational on the following VLANs:
50
Insertion of Option 82 is disabled
Verification of MAC address is enabled
DHCP snooping trust is configured on the following interfaces:
Interface             Trusted        Pkt Limit
------------          -------        ---------
Vethernet1            No              Unlimited
Vethernet2            Yes             Unlimited
Vethernet3            No              Unlimited
Vethernet4            No              Unlimited
Vethernet5            No              Unlimited
Vethernet6            No              Unlimited
Vethernet7            No              Unlimited

 I can see now that a binding table entry has been built:


DCNexus1000V# show ip dhcp snooping binding
MacAddress         IpAddress        LeaseSec  Type        VLAN  Interface
-----------------  ---------------  --------  ----------  ----  -------------
00:50:56:01:fa:ce  172.21.5.11      691129     dhcp-snoop  50    Vethernet7 

So you can see that IP DHCP Snooping protects me from a rogue DHCP server: if i don't trust the port, then no DHCP responses sent out that port will be trusted, they will be dropped.

DHCP Snooping also protects me from two other attacks that I had never even envisioned!

The first potential attack is taken care of by this default command:

DCNexus1000V# show run all | inc "ip dhcp snooping"

ip dhcp snooping
no ip dhcp snooping information option
ip dhcp snooping verify mac-addressip dhcp snooping vlan 50


This setting is set by default on the Nexus 1000V, what this does is check that when a DHCP request is sent, that the DHCP Client Hardware Address Value (Part of the DHCP packet) matches the source MAC Address, if it does not then the frame is dropped,this is to prevent an attacker running a DOS attack where by the attacker generates lots of DHCP requests in an attempt to empty the DHCP server of any available addresses in it's scope.

As you can imagine however depending on your configuration you may have some device that proxies DHCP requests on a clients behalf and thus with this command you must take that into account.

The other feature that DHCP performs is that if a DHCP release request is sent or a decline message, the switch checks its binding table to make sure that the dhcp release request is being sent from the same port where that IP is listed in it's binding table, to prevent attackers from sending a DHCP release request with your IP address to the DHCP server in the hope that when they then request an IP address, the DHCP server then allocates your address (which to be honest, seems fairly unlikely to me and quite a sophisticated attack!)


IP Source Guard

This feature does as advertised: if you have DHCP snooping enabled then the switch can ensure that the ip address coming out of the interface matches the one allocated via DHCP, by default it checks that both the MAC address and IP address match, but if your MAC address is likely to change, Nexus 1000V can just check that the SOURCE IP is correct.

DCNexus1000V(config)# ip source binding filter-mode ?
  ip      Source IP-filter only
  ip-mac  IP-mac filter

You can see the option to configure this above.

To actually specify this on an interface:

interface Vethernet7
  inherit port-profile DHCP_UNTRUSTED
  description DHCP_Windows2008_2, Network Adapter 1
  vmware dvport 289 dvswitch uuid "ff 19 20 50 81 e8 3a f3-40 ab f2 17 7c ff a3 95"
  vmware vm mac 0050.5601.FACE
  ip verify source dhcp-snooping-vlan

This is now configured on this interface, let's watch it in action.


So if we ping the DHCP we are able to with this command, but if we statically define the IP address on this host to 172.21.5.3 we are unable to ping

However if we define a static mapping:



DCNexus1000V(config)# ip source binding 172.21.5.3 0050.5601.face vlan 50 interface veth7


We are suddenly able to ping.

Now let's assume we enter in a binding but we enter a totally fake mac address, so only the IP matches:


DCNexus1000V(config)# ip source binding 172.21.5.3 face.face.face vlan 50 interface vethernet 7


If we left the configuration at this we would not be able to ping.

However if we change the config to verify IP address only and not mac address:


DCNexus1000V(config)# ip source binding filter-mode ip

Now we are able to ping because we are only verifying the IP address.