The Network Effect

Hi Guys

This is a very very quick blog post that I hope to expand in the future. (I apologise in advance for the disjointedness of this post.)

If like me you had huge problems getting the Nexus 1000v working at some points some troubleshooting tips would be lovely!!!

First of all, the most important troubleshooting you can do in my opinion is the vem-health command, you need to have console access to your ESXi server, here is a handy tip, you can do it via SSH, but if for some reason you only have console access, you can actually do it by pressing

Alt-F1

this will take you to the console.

Then, issue this command:

~ # vem-health check 00:50:56:a0:42:08
VSM Control MAC address: 00:50:56:a0:42:08
Control VLAN: 20
DPA MAC: 00:02:3d:40:03:02

The VEM-VSM connectivity seems to be fine.

How do you get this MAC address? go to your nexus 1000v and issue this command

Nexus1000v# show svs neighbors

Active Domain ID: 3

AIPC Interface MAC: 0050-56a0-4208
Inband Interface MAC: 0050-56a0-420a

The highlighted section is the MAC address you need.

Pay careful attention to what it says, if it says that the control is not working between the VEM and the VSM you need to investigate why.

Top gear top tip - to make things take affect, remove the vmware port group command and put it back in, note that this WILL remove it from your VMWARE configuration so you will need to add it back in.

Another two extremely useful commands:

~ # vemcmd show l2 20
Bridge domain    6 brtmax 4096, brtcnt 4, timeout 300
VLAN 20, swbd 20, ""
Flags: P - PVLAN S - Secure D - Drop
       Type         MAC Address   LTL   timeout   Flags    PVLAN
    Dynamic   00:50:56:a0:42:08   307         1
     Static   00:02:3d:80:03:02     6         0
    Dynamic   00:0c:29:e1:61:5a   307         1
     Static   00:02:3d:40:03:02    10         0

in the example above my control VLAN is VLAN 20. This allows me to see if any MAC addresses have been learnt out here, they should be! there should be some dynamic MAC's learnt if the control is working.

OK, next useful command:

~ # vemcmd show vlan 20
VLAN 20, vdc 1, swbd 20, hwbd 6, 3 ports
Portlist:
     10
     18 vmnic1
    307

this useful command helps you make sure that your control VLAN is actually bound to one of the interfaces.

I know this is a little rushed guys but I promise i will have a more detailed blog post on this later, i just needed to capture these two commands as they totally saved my bacon at one point!

Hi Guys!

if your anything like me you have had a bit of difficulty with the new NAT statements in the ASA 8.3 and above firmware, I have found it needs a little getting used to and thus I wanted to investigate and make sure I could document it and understand it for you guys :).

So, first of all let's have a look at some examples, in particular common things you would do on ASA pre 8.3

dynamic NAT overload
-------------------------

Easily the most common thing you might do on an ASA is NAT some internal addresses to a single global address, seems pretty common to me!

To give you some background I have two interfaces
Inside: 192.168.227.50
outside: 203.1.1.1

So, most likely we want to do something like say that anything from 192.168.227.0/24 is nat'd to our outside address, this would be accomplished on firmware below 8.3 with:

nat (inside) 1 192.168.227.0 255.255.255.0
global (outside) 1 interface

Here is the new way of accomplishing this.

first, you must specify an object that will define the NAT rules for this subnet, this is the way the ASA does NAT now so I am afraid we have to get used to it :p

object network insidenetwork
subnet 192.168.227.0 255.255.255.0
nat (inside,outside) dynamic interface
!

Now i will just mention, very confusingly these two statements when you do a show run will not be grouped together, confusing.

Anyway, so now with just that we should have enough to see an incoming nat connection when we ftp from our inside network to some host on the outside

ciscoasa# show xlate
1 in use, 1 most used
Flags: D - DNS, i - dynamic, r - portmap, s - static, I - identity, T - twice
TCP PAT from inside:192.168.227.40/1040 to outside:203.1.1.1/62977 flags ri idle 0:00:04 timeout 0:00:30

Success, we can see our NAT rule being applied.

Here is a handy way to see what NAT rule is being applied to a particular subnet:

ciscoasa# show nat 192.168.227.50 255.255.255.0 detail

Auto NAT Policies (Section 2)
1 (inside) to (outside) source dynamic insidenetwork interface
    translate_hits = 1, untranslate_hits = 0
    Source - Origin: 192.168.227.0/24, Translated: 203.1.1.1/24
Next let's look at what you might typically do, a static NAT.

Static NAT (Whole Host)
-------------------------

The new way of doing a static NAT is fairly similiar to the dynamic NAT we showed you previously, you must define an object that your going to NAT for

object network statichost
host 192.168.227.40
nat (inside,outside) static 203.1.1.253

Now we have a static NAT for that particular address range.

ciscoasa# show xlate
1 in use, 1 most used
Flags: D - DNS, i - dynamic, r - portmap, s - static, I - identity, T - twice
NAT from inside:192.168.227.40 to outside:203.1.1.253
    flags s idle 0:00:04 timeout 0:00:00

We can see more info from show nat detail:

ciscoasa# show nat detail

Auto NAT Policies (Section 2)
1 (inside) to (outside) source static statichost 203.1.1.253
    translate_hits = 3, untranslate_hits = 3
    Source - Origin: 192.168.227.40/32, Translated: 203.1.1.253/32
2 (inside) to (outside) source dynamic insidenetwork interface
    translate_hits = 1, untranslate_hits = 0
    Source - Origin: 192.168.227.0/24, Translated: 203.1.1.1/24

Static NAT (Individual Port)

-------------------------

Next let's have a look at static NAT for just a single port likeFTP

object network statichost
nat (inside,outside) static 203.1.1.253 service tcp ftp ftp

As usual this comes up for us as a xlate rule:

ciscoasa# show xlate
1 in use, 2 most used
Flags: D - DNS, i - dynamic, r - portmap, s - static, I - identity, T - twice
TCP PAT from inside:192.168.227.40 21-21 to outside:203.1.1.253 21-21
    flags sr idle 0:01:04 timeout 0:00:00
We can also see it under show nat detail:

ciscoasa# show nat detail

Auto NAT Policies (Section 2)
1 (inside) to (outside) source static statichost 203.1.1.253   service tcp ftp ftp
    translate_hits = 0, untranslate_hits = 2
    Source - Origin: 192.168.227.40/32, Translated: 203.1.1.253/32
    Service - Protocol: tcp Real: ftp Mapped: ftp
2 (inside) to (outside) source dynamic insidenetwork interface
    translate_hits = 1, untranslate_hits = 0
    Source - Origin: 192.168.227.0/24, Translated: 203.1.1.1/24

Intermission
----------------------
So far we are cooking with gas! and this probably all seems very familiar, the tricky parts are the next few parts, which includes things you do every day like making sure that VPN traffic is not Nat'd (Big one) and also making sure that your static translations that you have done work with VPN Clients (a common issue)

NAT Exemption
---------------------------
Let's have a look at how NAT exemption is performed.

Let's say you have a VPN on your outside interface, 10.0.0.0/24, and you want to exclude this from being NAT'd

The order of operation still matters in ASA 8.3 firmware, just like it did on your old ASA's, the only thing here is that there are three distinct sections, manual-section, the auto-section, and "After auto" manual section. (Confusing I know), as usual with NAT first match wins

So, first thing we have to do is create an object again (groan)

object network insidenetwork
subnet 192.168.227.0 255.255.255.0

object network vpnhosts
subnet 10.0.0.0 255.255.255.0

We already had insidenetwork defined but I am just showing it here again for your benefit :).

Next, you go into global config mode and specify that a NAT rule exists which nat's inside to itself and vpnhosts to itself.

nat (inside,outside) source static insidenetwork insidenetwork destination static vpnhosts vpnhosts

we can now see this in our show nat detail we have become so familiar with:
Manual NAT Policies (Section 1)
1 (inside) to (outside) source static insidenetwork insidenetwork   destination static vpnhosts vpnhosts
    translate_hits = 0, untranslate_hits = 0
    Source - Origin: 192.168.227.0/24, Translated: 192.168.227.0/24
    Destination - Origin: 10.0.0.0/24, Translated: 10.0.0.0/24

Auto NAT Policies (Section 2)
1 (inside) to (outside) source static statichost 203.1.1.253   service tcp ftp ftp
    translate_hits = 0, untranslate_hits = 0
    Source - Origin: 192.168.227.40/32, Translated: 203.1.1.253/32
    Service - Protocol: tcp Real: ftp Mapped: ftp
2 (inside) to (outside) source dynamic insidenetwork interface
    translate_hits = 7, untranslate_hits = 0
    Source - Origin: 192.168.227.0/24, Translated: 203.1.1.1/24

You will notice our NAT got put at the top, this is what we are expecting, and means that it will work the way we want it to :).

We then ping from a host in the 192.168.227.0 subnet

You can see that we now DON't have an entry for this in our NAT table:

ciscoasa# show xlate
2 in use, 4 most used
Flags: D - DNS, i - dynamic, r - portmap, s - static, I - identity, T - twice
NAT from inside:192.168.227.0/24 to outside:192.168.227.0/24
    flags sI idle 0:00:12 timeout 0:00:00
TCP PAT from inside:192.168.227.40 21-21 to outside:203.1.1.253 21-21
    flags sr idle 0:03:51 timeout 0:00:00

But if you look at the counter for show nat detail you can see that it is incrementing

ciscoasa# show nat detail
Manual NAT Policies (Section 1)
1 (inside) to (outside) source static insidenetwork insidenetwork   destination static vpnhosts vpnhosts
    translate_hits = 2, untranslate_hits = 3
    Source - Origin: 192.168.227.0/24, Translated: 192.168.227.0/24
    Destination - Origin: 10.0.0.0/24, Translated: 10.0.0.0/24
So now we have lots of NAT rules all working together super well :)

If you where to go to an outside host and try to ftp to the public address (203.1.1.253) that we setup, it would work, if you tried to communicate from the VPN (10.0.0.0) to a host in the 192.168.227.0 subnet, it would work, if you tried to get out to a public address from the host itself that would all work too.

More complicated example
------------------------------------

Let's go for a more complicated example, let's say you want to allow VPN users to VPN in and use your internet on the ASA, if you want to do this obviously you need to NAT the VPN as it goes back outside to the internet:

object network vpnhosts
nat (outside,outside) dynamic interface

Helpful Hints
-------------------

The NAT in ASA 8.3 can be a bit tricky at times, what helped me was using the
"show nat detail"
command, this command shows you all your rules, and you can watch as hits register against each of your NAT rules, it also shows you the order that the rules are applied in (order still matters just like it did in the previous ASA releases).

Finally, to see just the nat objects if you have a huge running config, I found the following command useful:

show run nat

I hope this helps someone out there

The Network Effect

VTP troubles with Pruning

Troubleshooting the Nexus 1000v

Reason number #18361 Why i hate juniper's

ASA 8.4 NAT, the (mostly) definitive guide

Running nested VM's

Popular old posts.