CCIE DC: Roll your own OTV

Hi Guys!

So today I played with OTV and managed to get it going on the new Cisco Cloud Services Router (CSR 1000V)

In this blog post I am going to show you the receipe I used to get this going :).

You will need:

• An ESX Server to run virtual images on
• Two guests on that VM running Windows or Linux (doesn't really matter)
• Two Cisco Cloud Services Router VM Guests
• Nexus 1000V and a VMWare DvSwitch, if no Nexus 1000V is available you can get away with just two DV-Switches :)

Basically, your going to have a topology that looks a little something like this:

Now if you haven't got the Cisco Cloud Services Router or you haven't set that up, which is really the first step you should check out the blog post from INE that explains how to configure it in VMWARE here.

OK, now that's set up. Let's quickly look at our topology.


So basically as you can see from the topology diagram our goal is to get the server 172.28.0.11 talking to 172.28.0.20 even though they are not in the same Layer 2 domain by using the CSR Routers to perform OTV for us :).

Let's quickly look at the initial configuration on the CSR Routers:

CSR 1:
interface GigabitEthernet1
 ip address 172.27.0.1 255.255.255.252
end

CSR 2:

interface GigabitEthernet1
 ip address 172.27.0.2 255.255.255.252
end


Can we ping across these hosts?

CSR1#ping 172.27.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.27.0.2, timeout is 2 seconds:
!!!!!

Yes we can, so now we get onto the next stage.


The next stage involves configuring our dvSwitch and our Nexus 1000V, if we are using the Nexus 1000V basically I create a port profile that trunks up the VLAN to my CSR:

port-profile type vethernet EXTEND_SITE_TRUNK
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 499-501
  no shutdown
  state enabled


VLAN 499 is my VLAN that the server will sit on, so I have created a port profile for that:


port-profile type vethernet EXTEND_VLAN_SERVER_1
  vmware port-group
  switchport access vlan 499
  switchport mode access
  no shutdown
  state enabled

Now my server and my CSR #1 Router can see each other once I assign the port profiles to the appropriate VM's.


Ok now we have to duplicate the same sort of thing on the dvSwitch, because in this example we are trying to make the two hosts be on totally seperate switches that are not connected in any way. To do this I created a DVSwitch with No Uplinks and assigned from there:












As you can see I set the server to use the untagged port group and the Router to use the Trunk Port Group, the reasoning for this will become clear later.


Ok now I have all the basic infrastructure setup, let's get onto the OTV Config.

First thing I need to do on both CSR's is specify a site identifier and a site VLAN (Or in the case of CSR, which is not like configuring OTV on a nexus 7000, I specify a Bridge-domain)

CSR1(config)#otv site bridge-domain 1
CSR1(config-otv-site)#exit
CSR1(config)#otv site-identifier 0x2

And then the same on CSR2 but with a diffirent site Identifier


CSR2(config)#otv site bridge-domain 1
CSR2(config-otv-site)#exit
CSR2(config)#otv site-identifier 0x1


Now we need to configure the actual OTV Interfaces, the first thing to do is go to our join interfaces (which are the 172.27.0.0/30 links we setup previously) and specify IGMP Version 3 to support SSM multicast

CSR1(config)#interface GigabitEthernet1
CSR1(config-if)# ip igmp version 3
!

Now we repeat the above on CSR2 and then we are ready to configure the OTV Interface:

CSR 1:
interface Overlay1
 no ip address
 otv join-interface GigabitEthernet1
 otv use-adjacency-server 172.27.0.1 unicast-only
 otv adjacency-server unicast-only
!

CSR 2:


interface Overlay1
 no ip address
 otv join-interface GigabitEthernet1
 otv use-adjacency-server 172.27.0.1 unicast-only
!

In the above example I have used an adjacency server that lives on CSR, by default OTV wants to use multicast but I had trouble getting this going with multicast, I believe the issue is related to the way I have the CSR's join interfaces connected to each other, so I did this to make life easier for myself for now.

As you can see one router (CSR1) is the adjacency server and I have configured CSR2 to point to this, you can have multiple adjacency servers specified for redundancy if you like.

OK now we need to configure the "extend" interface on our inside of our CSR, which is the interface on VLAN 499 on each switch respectively that heads down towards our end host servers.

Here is the configuration there:

interface GigabitEthernet2
 no ip address
 negotiation auto

 service instance 2 ethernet
  encapsulation dot1q 499
  bridge-domain 200
 !
end


I believe that the CSR has the same restriction that a Nexus 7000 has in terms of you can't have IP addresses assigned to interfaces that your trying to extend over the OTV. If anyone knows for sure if this is possible/not possible please post in the comments section :).

So the most important part of this configuration is where we specify a service instance, this will be how we map our OTV VLAN, as you can see the VLAN number is set here as well as a bridge domain.

If we now go back to our OTV interface:

interface Overlay1
 no ip address
 otv join-interface GigabitEthernet1
 otv use-adjacency-server 172.27.0.1 unicast-only
 service instance 2 ethernet
  encapsulation dot1q 499
  bridge-domain 200
  no shutdown
!
 
end

 Here we are actually mapping that service instance we created, now when we perform a no shut we should see the OTV tunnel come up, here are some useful troubleshooting commands to verify:


CSR1#show otv detail
Overlay Interface Overlay1
 VPN name                 : None
 VPN ID                   : 1
 State                    : UP
 AED Capable              : Yes
 Join interface(s)        : GigabitEthernet1
 Join IPv4 address        : 172.27.0.1
 Tunnel interface(s)      : Tunnel0
 Encapsulation format     : GRE/IPv4
 Site Bridge-Domain       : 1
 Capability               : Unicast-only
 Is Adjacency Server      : Yes
 Adj Server Configured    : Yes
 Prim/Sec Adj Svr(s)      : 172.27.0.1
 OTV instance(s)          : 0
 FHRP Filtering Enabled   : Yes
 ARP Suppression Enabled  : Yes
 ARP Cache Timeout        : 600 seconds


This first command shows us that the AED has been elected (it's us), that our join interface is Gi1 and that our bridge ID is 1, also that we are using an adjacency server and that we ourselves are configured as an adjacency server. If you have errors in here you should revisit your config and make sure your not missing any of the critical commands such as the site bridge domain.

The next useful troubleshooting command is to establish if an ISIS adjacency exists between the hosts:


CSR1#show otv isis neighbors

Tag Overlay1:
System Id      Type Interface   IP Address      State Holdtime Circuit Id
CSR2           L1   Ov1         172.27.0.2      UP    25       CSR1.01 

If the OTV is successful the only state for this should be UP, you will also notice that you can see the hostname of your peer.

CSR1#show otv adjacency
Overlay 1 Adjacency Database
Hostname                       System-ID      Dest Addr       Up Time   State
CSR2                           001e.4925.fc00 172.27.0.2      00:20:11  UP 
!

Here is another helpful command that just helps you verify the connectivity.

So far, it looks like everything is up and we should be good to go, let's do a ping from our host!








Success! Our OTV Tunnel is carrying the traffic, we can do some verification here too (this will also be helpful to you when troubleshooting if it does not work)




CSR1#show otv route

Codes: BD - Bridge-Domain, AD - Admin-Distance,
       SI - Service Instance, * - Backup Route

OTV Unicast MAC Routing Table for Overlay1

 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
----------------------------------------------------------
 0    499  200    0050.5601.face 50    ISIS   CSR2
 0    499  200    0050.56a0.1aad 40    BD Eng Gi2:SI2


As you can see this shows some very useful information, you can see that the above routes are in the ISIS routing table, and what bridge-domain and VLAN they are attached to, also the owner helps you know if its local to you or over the OTV tunnel.

You can also see that it knows that one of the Mac's is actually pointed out interface Gi2:SI2 (Gigabit 2, Service Instance 2)


So this is quite a helpful command. Another useful command is:

CSR1#show otv arp-nd-cache
Overlay1 ARP/ND L3->L2 Address Mapping Cache
BD     MAC            Layer-3 Address  Age (HH:MM:SS) Local/Remote
200    0050.5601.face 172.28.0.11      00:02:46       Remote

This will show you the ARP cache that the router will respond with whenever it see's an ARP request, this is part of the proxy-arp function that OTV performs to ensure that ARP flooding is not occuring too often, this would be a good place to look if you had recently changed IP address or recently changed a host on the OTV and where wondering why you couldn't see it, it could be that the ARP cache is out of date.


Finally, if your using the Nexus 1000V you can also use the following helpful troubleshooting command:

DCNexus1000V# show mac address-table vlan 499
VLAN      MAC Address       Type    Age       Port                           Mod
---------+-----------------+-------+---------+------------------------------+---
499       0050.56a0.115e    static  0         Veth11                         3 
499       0050.56a0.1aad    static  0         Veth2                          3 
499       001e.bd93.82bc    dynamic 15        Veth11                         3 
499       0050.5601.face    dynamic 248       Veth11                         3



So Veth11 is the interface on my Nexus 1000V that goes towards my CSR router, so the fact that I have learnt the mac 0050.5601.face via this interface shows that the traffic is coming over the OTV.


I hope this helps someone out there, especially my fellow CCIE DC candidates.