Hi Guys!
Final blog post for tonight I have spent lots of time this weekend working super hard on my study!
So let's check out FCoE NPV, first a quick word of advice: Make sure when switching fibre channel modes (NPV vs FC Switching) make sure you completely write erase before you switch modes.
Without further adue, let's look at the config!
First thing's first, let's turn on the FCoE NPV Feature on the switch that is going to be on the actual switch performing NPV:
feature fcoe-npv
On the FC switch that will act as our core and WILL participate in the FC as a FC Forwarder, we need to turn on NPIV
feature npiv
As usual with FCoE, let's create our VSAN's and our VLAN's
vsan database
vsan 10
!
vlan 10
fcoe vsan 10
!
Next, we configure our ethernet interface that faces up towards the core, we can do this on both our core switch and our NPV Switch
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10,20
!
Next, let's configure our VFC Interface:
interface vfc1
bind interface Ethernet1/10
switchport mode NP
switchport trunk mode on
no shutdown
This is now ready on our NPV side, on our Core switch the config is only slightly diffirent:
interface vfc1
bind interface Ethernet1/10
switchport trunk allowed vsan 10
no shutdown
As you can see on the upstream switch our core switch just has this configured as an F Port.
Now finally, we need to make each of them realise that the port is a member of the VSAN:
vsan database
vsan 10 interface vfc1
Now we can check to see if the vfc interface is up on our NPV Switch:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is NP, trunk mode is on
snmp link state traps are enabled
Port mode is TNP
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
Success! the next step is to get our server facing interface going, for the sake of brevity i will show all the configuration for this at once:
switch# show run int vfc2
interface vfc2
bind interface Ethernet1/1
switchport trunk mode on
no shutdown
!
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan 10,20
spanning-tree port type edge trunk
!
vsan database
vsan 10 interface vfc2
Let's check the server facing interface:
switch# show int vfc2
vfc2 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/1
Hardware is Ethernet
Port WWN is 20:01:54:7f:ee:af:1c:bf
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
!
Success! Let's check the NPV flogi table which is very useful.
switch# show npv flogi-table
--------------------------------------------------------------------------------
SERVER EXTERNAL INTERFACE VSAN FCID PORT NAME NODE NAME INTERFACE
--------------------------------------------------------------------------------
vfc2 10 0xd80002 20:00:a4:4c:11:13:8c:d1 10:00:a4:4c:11:13:8c:d1 vfc1
Very Useful output showing us that the port vfc2 is logged in and is using vfc1 for its external interface.
The show npv command is actually quite powerful:
switch# show npv ?
external-interface-usage Show external interface usage by server interfaces
flogi-table Show information about FLOGI sessions
internal Show internal NPV information
status Show NPV status
traffic-map Show information about Traffic Map
In particular, i found the following command VERY helpful:
in parti
switch# show npv internal events
1) Event:E_DEBUG, length:94, at 270505 usecs after Sun Apr 21 10:39:22 2013
[538976288] E(10,vfc1) Received GMAL Response from core switch with Core Switch Inet Addr: 10.1.1.15
2) Event:E_DEBUG, length:252, at 270497 usecs after Sun Apr 21 10:39:22 2013
[538976288] E(10,vfc1) npivp_get_ext_intf_fsm(1466): Is Core VF Capable: TRU
E, Is Phy Login Done: TRUE, Is Port Channel: FALSE RID: { Type: Ext-If-VSAN(8),
IfIndex: vfc1(0x1e000000), UCD: 0, VSAN: 10, PWWN: 00:00:00:00:00:00:00:00 }, St
ate NPIVP_EXT_IF_ST_UP
This, along with show npv internal errors is really helpful in troubleshooting upstream issues.
If for example the upstream switch was not enabled for NPIV, I would see the following output in this command:
11) Event:E_DEBUG, length:98, at 464269 usecs after Sun Apr 21 10:41:22 2013
[112] E(10,vfc1) FC Upstream switch is not NPIV enabled. Bringing down the external interface: vfc1
CCIE DC: Multihop FCoE on Nexus 5k
Hi Guys.
There is a lot of confusion in my own mind for FCoE Multihop, I remember back in the day that the argument was "well, you can do FCoE with the Nexus 2k's which is not true multihop but it kind of is!" I also remember that this blog post: http://brasstacksblog.typepad.com/brass-tacks/2011/06/fcfcoe-connectivity-options-as-of-6272011.html by Erik was one of the best posts on the topic. I have asked Erik Via Twitter to update this diagram as a lot of it has changed now in regards to Cisco UCS.
In this blog post I will show one of the examples given by Erik, which is multihop FCOE Configuration on the Nexus 5k's in the hope that it will help someone out there.
Authors note: I think that the order of operation is very important when configuring multihop FCoE as I had major issues getting this going the first time I attempted it. My advice is to get the absolute basic FCoE Connectivity going first, then add complexity once your comfortable that it's all configured correctly.
Steps:
After enabling FCoE Ofcourse with feature FCOE on your nexus 5k,The first thing to do on both your Nexus 5k's is configure the appropriate VLAN's and VSAN's.
vsan database
vsan 10
!
vlan 10
fcoe vsan 10
vlan 20
name DataVLAN
In this example I have configured VSAN 10, and bound it to VLAN 10.
Next, I need to configure the ethernet interface between my two switches appropriately:
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10
!
At this point I personally still don't "no shut" the interfaces and I wait until I have configured everything.
The next step is to configure the VFC Interface:
interface vfc1
bind interface Ethernet1/10
switchport mode E
switchport trunk allowed vsan 10
no shutdown
!
Now this is done, I no shut the ethernet interface on each end:
int eth1/10
no shut
!
Now let's look at our VFC1 interface:
switch# show int vfc1
vfc1 is trunking
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 216 bits/sec, 27 bytes/sec, 0 frames/sec
1 minute output rate 200 bits/sec, 25 bytes/sec, 0 frames/sec
3667 frames input, 440724 bytes
0 discards, 0 errors
3790 frames output, 521116 bytes
0 discards, 0 errors
last clearing of "show interface" counters Sun Apr 21 04:54:33 2013
Interface last changed at Sun Apr 21 05:34:21 2013
Success! Our VSAN is trunking across the link, a show fcns database helps us verify this:
switch# show fcns database
VSAN 10:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0f08d1 NL 21:00:00:0a:60:55:17:69 (Seagate) scsi-fcp:target
[disk]
0x0f08d2 NL 21:00:00:11:d6:3e:ee:2e scsi-fcp:target
[disk1]
Finally an FCPING helps us verify 100 percent.
switch# fcping pwwn 22:00:00:04:cf:21:a5:2e vsan 10
28 bytes from 22:00:00:04:cf:21:a5:2e time = 1759 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 287 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 222 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 304 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 270 usec
At this point we have an FCoE Multihop topology.
If all you came for is how to configure an FCoE Trunk between the two 5k's you can stop reading now as now we are getting into a bit more detail
So the first thing I asked myself is, does this VLAN run spanning tree?
ToSanSWITCH# show spanning-tree vlan 10
Spanning tree instance(s) for vlan does not exist.
Alright, that answers that question, but what happens if i assign a normal ethernet port to this FCoE VLAN?:
ToSanSWITCH(config)# int eth1/1
ToSanSWITCH(config-if)# switchport access vlan 10
ToSanSWITCH(config-if)# end
What does my show spanning-tree say now?
ToSanSWITCH# show spanning-tree vlan 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32778
Address 547f.eeaf.3a3c
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 547f.eeaf.3a3c
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/1 Desg FWD 2 128.129 P2p
So according to this, the VLAN is forwarding on Eth1/1, at first I was quite confused and wondered to myself what kind of effect this would have on the FCoE Traffic, but it occured to me that the ethertype for FCoE Traffic is diffirent, what I imagine would happen is that Eth1/1 would receive copies of the FCoE Frames, but if the device plugged in at Eth1/1 has no idea what those frames are (i.e. doesn't recognise or want to use the ethertype) chances are it will just ignore them.
Regardless, this configuration had no ill effect on my VFC Interface as it was still showing as up:
ToSanSWITCH# show int vfc1
vfc1 is trunking
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:3a:3f
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
I decided to see if I could convince it to "break", the first thing I tried was adding a VSAN to this interface that was not configured with an appropriate FCoE VLAN, which in this case was VSAN 1
I configured this on both ends:
switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1
The worst that happens is that the VSAN 1 stays in isolation mode:
switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1
Next, I configured on one of the switches a port facing a server:
interface Ethernet1/1
switchport mode trunk
switchport trunk native vlan 20
spanning-tree port type edge trunk
!
interface vfc10
bind interface Ethernet1/1
switchport trunk allowed vsan 10
no shutdown
!
vsan database
vsan 10 interface vfc10
!
The server port comes up fine:
switch# show int vfc10
vfc10 is trunking
Bound interface is Ethernet1/1
Hardware is Ethernet
Port WWN is 20:09:54:7f:ee:af:1c:bf
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 10
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
At this point I am determined to make it my mission to break this FCoE Link as when I first tried to set this up, I had major problems, I want to see what configuration or what order of configuration you have to do for it NOT to work.
Some of you may have noticed that despite what the cisco configuration guide mentions I did NOT have to add these trunk interfaces to my vsan database, i.e. i did NOT need the following configuration:
vsan database
vsan 10 interface vfc1
I decide to add this config to see if this kills the VFC, it does not:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
So at this point I have not been able to get the VFC not to work for VSAN 10, it does not work for VSAN 1 but that is because I do not have an equivilant FCoE VLAN for that VSAN, which I can totally understand.
Next I try making the Ethernet interface carry both LAN and SAN Traffic:
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10,20
switch# show spanning-tree int eth1/10
Vlan Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0020 Desg FWD 2 128.138 P2p
This does not faze the device and the VFC interface remains up:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
At this point I decide to up the ante by adding another interface:
interface Ethernet1/20
switchport mode trunk
switchport trunk allowed vlan 10,20
!
interface vfc2
bind interface Ethernet1/20
switchport mode E
no shutdown
!
This new interface comes up just fine:
switch# show int vfc2
vfc2 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/20
Hardware is Ethernet
Port WWN is 20:01:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) (1)
I even get nice load balancing across the links:
switch# show fspf database vsan 10
FSPF Link State Database for VSAN 10 Domain 0xd8(216)
LSR Type = 1
Advertising domain ID = 0xd8(216)
LSR Age = 341
LSR Incarnation number = 0x80000013
LSR Checksum = 0x79e0
Number of links = 3
NbrDomainId IfIndex NbrIfIndex Link Type Cost
-----------------------------------------------------------------------------
0x0f(15) 0x00040000 0x00040000 1 125
0x9a(154) 0x001e0000 0x001e0000 1 100
0x9a(154) 0x001e0001 0x001e0001 1 100
So at this point, everything is working pretty well. I decide to try making the link between the two switches a port-channel:
feature lacp
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10,20
channel-group 20 mode active
!
interface Ethernet1/20
switchport mode trunk
switchport trunk allowed vlan 10,20
channel-group 20 mode active
!
interface vfc1
bind interface port-channel20
switchport mode E
switchport trunk allowed vsan 10
no shutdown
!
The VFC Int is still up and carrying traffic:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is port-channel10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
At this point I don't understand why I had trouble getting the FCoE VE Int to come up the first time but am willing to accept that the feature works perfectly and it must have been some sort of order of operation issue.
There is a lot of confusion in my own mind for FCoE Multihop, I remember back in the day that the argument was "well, you can do FCoE with the Nexus 2k's which is not true multihop but it kind of is!" I also remember that this blog post: http://brasstacksblog.typepad.com/brass-tacks/2011/06/fcfcoe-connectivity-options-as-of-6272011.html by Erik was one of the best posts on the topic. I have asked Erik Via Twitter to update this diagram as a lot of it has changed now in regards to Cisco UCS.
In this blog post I will show one of the examples given by Erik, which is multihop FCOE Configuration on the Nexus 5k's in the hope that it will help someone out there.
Authors note: I think that the order of operation is very important when configuring multihop FCoE as I had major issues getting this going the first time I attempted it. My advice is to get the absolute basic FCoE Connectivity going first, then add complexity once your comfortable that it's all configured correctly.
Steps:
After enabling FCoE Ofcourse with feature FCOE on your nexus 5k,The first thing to do on both your Nexus 5k's is configure the appropriate VLAN's and VSAN's.
vsan database
vsan 10
!
vlan 10
fcoe vsan 10
vlan 20
name DataVLAN
In this example I have configured VSAN 10, and bound it to VLAN 10.
Next, I need to configure the ethernet interface between my two switches appropriately:
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10
!
At this point I personally still don't "no shut" the interfaces and I wait until I have configured everything.
The next step is to configure the VFC Interface:
interface vfc1
bind interface Ethernet1/10
switchport mode E
switchport trunk allowed vsan 10
no shutdown
!
Now this is done, I no shut the ethernet interface on each end:
int eth1/10
no shut
!
Now let's look at our VFC1 interface:
switch# show int vfc1
vfc1 is trunking
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
1 minute input rate 216 bits/sec, 27 bytes/sec, 0 frames/sec
1 minute output rate 200 bits/sec, 25 bytes/sec, 0 frames/sec
3667 frames input, 440724 bytes
0 discards, 0 errors
3790 frames output, 521116 bytes
0 discards, 0 errors
last clearing of "show interface" counters Sun Apr 21 04:54:33 2013
Interface last changed at Sun Apr 21 05:34:21 2013
Success! Our VSAN is trunking across the link, a show fcns database helps us verify this:
switch# show fcns database
VSAN 10:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x0f08d1 NL 21:00:00:0a:60:55:17:69 (Seagate) scsi-fcp:target
[disk]
0x0f08d2 NL 21:00:00:11:d6:3e:ee:2e scsi-fcp:target
[disk1]
Finally an FCPING helps us verify 100 percent.
switch# fcping pwwn 22:00:00:04:cf:21:a5:2e vsan 10
28 bytes from 22:00:00:04:cf:21:a5:2e time = 1759 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 287 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 222 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 304 usec
28 bytes from 22:00:00:04:cf:21:a5:2e time = 270 usec
At this point we have an FCoE Multihop topology.
If all you came for is how to configure an FCoE Trunk between the two 5k's you can stop reading now as now we are getting into a bit more detail
So the first thing I asked myself is, does this VLAN run spanning tree?
ToSanSWITCH# show spanning-tree vlan 10
Spanning tree instance(s) for vlan does not exist.
Alright, that answers that question, but what happens if i assign a normal ethernet port to this FCoE VLAN?:
ToSanSWITCH(config)# int eth1/1
ToSanSWITCH(config-if)# switchport access vlan 10
ToSanSWITCH(config-if)# end
What does my show spanning-tree say now?
ToSanSWITCH# show spanning-tree vlan 10
VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32778
Address 547f.eeaf.3a3c
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 547f.eeaf.3a3c
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/1 Desg FWD 2 128.129 P2p
So according to this, the VLAN is forwarding on Eth1/1, at first I was quite confused and wondered to myself what kind of effect this would have on the FCoE Traffic, but it occured to me that the ethertype for FCoE Traffic is diffirent, what I imagine would happen is that Eth1/1 would receive copies of the FCoE Frames, but if the device plugged in at Eth1/1 has no idea what those frames are (i.e. doesn't recognise or want to use the ethertype) chances are it will just ignore them.
Regardless, this configuration had no ill effect on my VFC Interface as it was still showing as up:
ToSanSWITCH# show int vfc1
vfc1 is trunking
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:3a:3f
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
I decided to see if I could convince it to "break", the first thing I tried was adding a VSAN to this interface that was not configured with an appropriate FCoE VLAN, which in this case was VSAN 1
I configured this on both ends:
switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1
The worst that happens is that the VSAN 1 stays in isolation mode:
switch(config)# int vfc1
switch(config-if)# switchport trunk allowed vsan add 1
Next, I configured on one of the switches a port facing a server:
interface Ethernet1/1
switchport mode trunk
switchport trunk native vlan 20
spanning-tree port type edge trunk
!
interface vfc10
bind interface Ethernet1/1
switchport trunk allowed vsan 10
no shutdown
!
vsan database
vsan 10 interface vfc10
!
The server port comes up fine:
switch# show int vfc10
vfc10 is trunking
Bound interface is Ethernet1/1
Hardware is Ethernet
Port WWN is 20:09:54:7f:ee:af:1c:bf
Admin port mode is F, trunk mode is on
snmp link state traps are enabled
Port mode is TF
Port vsan is 10
Trunk vsans (admin allowed and active) (10)
Trunk vsans (up) (10)
At this point I am determined to make it my mission to break this FCoE Link as when I first tried to set this up, I had major problems, I want to see what configuration or what order of configuration you have to do for it NOT to work.
Some of you may have noticed that despite what the cisco configuration guide mentions I did NOT have to add these trunk interfaces to my vsan database, i.e. i did NOT need the following configuration:
vsan database
vsan 10 interface vfc1
I decide to add this config to see if this kills the VFC, it does not:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
So at this point I have not been able to get the VFC not to work for VSAN 10, it does not work for VSAN 1 but that is because I do not have an equivilant FCoE VLAN for that VSAN, which I can totally understand.
Next I try making the Ethernet interface carry both LAN and SAN Traffic:
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10,20
switch# show spanning-tree int eth1/10
Vlan Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
VLAN0020 Desg FWD 2 128.138 P2p
This does not faze the device and the VFC interface remains up:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
At this point I decide to up the ante by adding another interface:
interface Ethernet1/20
switchport mode trunk
switchport trunk allowed vlan 10,20
!
interface vfc2
bind interface Ethernet1/20
switchport mode E
no shutdown
!
This new interface comes up just fine:
switch# show int vfc2
vfc2 is trunking (Not all VSANs UP on the trunk)
Bound interface is Ethernet1/20
Hardware is Ethernet
Port WWN is 20:01:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
Trunk vsans (initializing) (1)
I even get nice load balancing across the links:
switch# show fspf database vsan 10
FSPF Link State Database for VSAN 10 Domain 0xd8(216)
LSR Type = 1
Advertising domain ID = 0xd8(216)
LSR Age = 341
LSR Incarnation number = 0x80000013
LSR Checksum = 0x79e0
Number of links = 3
NbrDomainId IfIndex NbrIfIndex Link Type Cost
-----------------------------------------------------------------------------
0x0f(15) 0x00040000 0x00040000 1 125
0x9a(154) 0x001e0000 0x001e0000 1 100
0x9a(154) 0x001e0001 0x001e0001 1 100
So at this point, everything is working pretty well. I decide to try making the link between the two switches a port-channel:
feature lacp
interface Ethernet1/10
switchport mode trunk
switchport trunk allowed vlan 10,20
channel-group 20 mode active
!
interface Ethernet1/20
switchport mode trunk
switchport trunk allowed vlan 10,20
channel-group 20 mode active
!
interface vfc1
bind interface port-channel20
switchport mode E
switchport trunk allowed vsan 10
no shutdown
!
The VFC Int is still up and carrying traffic:
switch# show int vfc1
vfc1 is trunking (Not all VSANs UP on the trunk)
Bound interface is port-channel10
Hardware is Ethernet
Port WWN is 20:00:54:7f:ee:af:1c:bf
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 10
Trunk vsans (admin allowed and active) (1,10)
Trunk vsans (up) (10)
Trunk vsans (isolated) ()
At this point I don't understand why I had trouble getting the FCoE VE Int to come up the first time but am willing to accept that the feature works perfectly and it must have been some sort of order of operation issue.
CCIE DC: Roll your own OTV
Hi Guys!
So today I played with OTV and managed to get it going on the new Cisco Cloud Services Router (CSR 1000V)
In this blog post I am going to show you the receipe I used to get this going :).
You will need:
Basically, your going to have a topology that looks a little something like this:
Now if you haven't got the Cisco Cloud Services Router or you haven't set that up, which is really the first step you should check out the blog post from INE that explains how to configure it in VMWARE here.
OK, now that's set up. Let's quickly look at our topology.
So basically as you can see from the topology diagram our goal is to get the server 172.28.0.11 talking to 172.28.0.20 even though they are not in the same Layer 2 domain by using the CSR Routers to perform OTV for us :).
Let's quickly look at the initial configuration on the CSR Routers:
CSR 1:
interface GigabitEthernet1
ip address 172.27.0.1 255.255.255.252
end
CSR 2:
interface GigabitEthernet1
ip address 172.27.0.2 255.255.255.252
end
Can we ping across these hosts?
CSR1#ping 172.27.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.27.0.2, timeout is 2 seconds:
!!!!!
Yes we can, so now we get onto the next stage.
The next stage involves configuring our dvSwitch and our Nexus 1000V, if we are using the Nexus 1000V basically I create a port profile that trunks up the VLAN to my CSR:
port-profile type vethernet EXTEND_SITE_TRUNK
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 499-501
no shutdown
state enabled
VLAN 499 is my VLAN that the server will sit on, so I have created a port profile for that:
port-profile type vethernet EXTEND_VLAN_SERVER_1
vmware port-group
switchport access vlan 499
switchport mode access
no shutdown
state enabled
Now my server and my CSR #1 Router can see each other once I assign the port profiles to the appropriate VM's.
Ok now we have to duplicate the same sort of thing on the dvSwitch, because in this example we are trying to make the two hosts be on totally seperate switches that are not connected in any way. To do this I created a DVSwitch with No Uplinks and assigned from there:
As you can see I set the server to use the untagged port group and the Router to use the Trunk Port Group, the reasoning for this will become clear later.
Ok now I have all the basic infrastructure setup, let's get onto the OTV Config.
First thing I need to do on both CSR's is specify a site identifier and a site VLAN (Or in the case of CSR, which is not like configuring OTV on a nexus 7000, I specify a Bridge-domain)
CSR1(config)#otv site bridge-domain 1
CSR1(config-otv-site)#exit
CSR1(config)#otv site-identifier 0x2
And then the same on CSR2 but with a diffirent site Identifier
CSR2(config)#otv site bridge-domain 1
CSR2(config-otv-site)#exit
CSR2(config)#otv site-identifier 0x1
Now we need to configure the actual OTV Interfaces, the first thing to do is go to our join interfaces (which are the 172.27.0.0/30 links we setup previously) and specify IGMP Version 3 to support SSM multicast
CSR1(config)#interface GigabitEthernet1
CSR1(config-if)# ip igmp version 3
!
Now we repeat the above on CSR2 and then we are ready to configure the OTV Interface:
CSR 1:
interface Overlay1
no ip address
otv join-interface GigabitEthernet1
otv use-adjacency-server 172.27.0.1 unicast-only
otv adjacency-server unicast-only
!
CSR 2:
interface Overlay1
no ip address
otv join-interface GigabitEthernet1
otv use-adjacency-server 172.27.0.1 unicast-only
!
In the above example I have used an adjacency server that lives on CSR, by default OTV wants to use multicast but I had trouble getting this going with multicast, I believe the issue is related to the way I have the CSR's join interfaces connected to each other, so I did this to make life easier for myself for now.
As you can see one router (CSR1) is the adjacency server and I have configured CSR2 to point to this, you can have multiple adjacency servers specified for redundancy if you like.
OK now we need to configure the "extend" interface on our inside of our CSR, which is the interface on VLAN 499 on each switch respectively that heads down towards our end host servers.
Here is the configuration there:
interface GigabitEthernet2
no ip address
negotiation auto
service instance 2 ethernet
encapsulation dot1q 499
bridge-domain 200
!
end
I believe that the CSR has the same restriction that a Nexus 7000 has in terms of you can't have IP addresses assigned to interfaces that your trying to extend over the OTV. If anyone knows for sure if this is possible/not possible please post in the comments section :).
So the most important part of this configuration is where we specify a service instance, this will be how we map our OTV VLAN, as you can see the VLAN number is set here as well as a bridge domain.
If we now go back to our OTV interface:
interface Overlay1
no ip address
otv join-interface GigabitEthernet1
otv use-adjacency-server 172.27.0.1 unicast-only
service instance 2 ethernet
encapsulation dot1q 499
bridge-domain 200
no shutdown
!
end
Here we are actually mapping that service instance we created, now when we perform a no shut we should see the OTV tunnel come up, here are some useful troubleshooting commands to verify:
CSR1#show otv detail
Overlay Interface Overlay1
VPN name : None
VPN ID : 1
State : UP
AED Capable : Yes
Join interface(s) : GigabitEthernet1
Join IPv4 address : 172.27.0.1
Tunnel interface(s) : Tunnel0
Encapsulation format : GRE/IPv4
Site Bridge-Domain : 1
Capability : Unicast-only
Is Adjacency Server : Yes
Adj Server Configured : Yes
Prim/Sec Adj Svr(s) : 172.27.0.1
OTV instance(s) : 0
FHRP Filtering Enabled : Yes
ARP Suppression Enabled : Yes
ARP Cache Timeout : 600 seconds
This first command shows us that the AED has been elected (it's us), that our join interface is Gi1 and that our bridge ID is 1, also that we are using an adjacency server and that we ourselves are configured as an adjacency server. If you have errors in here you should revisit your config and make sure your not missing any of the critical commands such as the site bridge domain.
The next useful troubleshooting command is to establish if an ISIS adjacency exists between the hosts:
CSR1#show otv isis neighbors
Tag Overlay1:
System Id Type Interface IP Address State Holdtime Circuit Id
CSR2 L1 Ov1 172.27.0.2 UP 25 CSR1.01
If the OTV is successful the only state for this should be UP, you will also notice that you can see the hostname of your peer.
CSR1#show otv adjacency
Overlay 1 Adjacency Database
Hostname System-ID Dest Addr Up Time State
CSR2 001e.4925.fc00 172.27.0.2 00:20:11 UP
!
Here is another helpful command that just helps you verify the connectivity.
So far, it looks like everything is up and we should be good to go, let's do a ping from our host!
Success! Our OTV Tunnel is carrying the traffic, we can do some verification here too (this will also be helpful to you when troubleshooting if it does not work)
CSR1#show otv route
Codes: BD - Bridge-Domain, AD - Admin-Distance,
SI - Service Instance, * - Backup Route
OTV Unicast MAC Routing Table for Overlay1
Inst VLAN BD MAC Address AD Owner Next Hops(s)
----------------------------------------------------------
0 499 200 0050.5601.face 50 ISIS CSR2
0 499 200 0050.56a0.1aad 40 BD Eng Gi2:SI2
As you can see this shows some very useful information, you can see that the above routes are in the ISIS routing table, and what bridge-domain and VLAN they are attached to, also the owner helps you know if its local to you or over the OTV tunnel.
You can also see that it knows that one of the Mac's is actually pointed out interface Gi2:SI2 (Gigabit 2, Service Instance 2)
So this is quite a helpful command. Another useful command is:
CSR1#show otv arp-nd-cache
Overlay1 ARP/ND L3->L2 Address Mapping Cache
BD MAC Layer-3 Address Age (HH:MM:SS) Local/Remote
200 0050.5601.face 172.28.0.11 00:02:46 Remote
This will show you the ARP cache that the router will respond with whenever it see's an ARP request, this is part of the proxy-arp function that OTV performs to ensure that ARP flooding is not occuring too often, this would be a good place to look if you had recently changed IP address or recently changed a host on the OTV and where wondering why you couldn't see it, it could be that the ARP cache is out of date.
Finally, if your using the Nexus 1000V you can also use the following helpful troubleshooting command:
DCNexus1000V# show mac address-table vlan 499
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---
499 0050.56a0.115e static 0 Veth11 3
499 0050.56a0.1aad static 0 Veth2 3
499 001e.bd93.82bc dynamic 15 Veth11 3
499 0050.5601.face dynamic 248 Veth11 3
So Veth11 is the interface on my Nexus 1000V that goes towards my CSR router, so the fact that I have learnt the mac 0050.5601.face via this interface shows that the traffic is coming over the OTV.
I hope this helps someone out there, especially my fellow CCIE DC candidates.
So today I played with OTV and managed to get it going on the new Cisco Cloud Services Router (CSR 1000V)
In this blog post I am going to show you the receipe I used to get this going :).
You will need:
- An ESX Server to run virtual images on
- Two guests on that VM running Windows or Linux (doesn't really matter)
- Two Cisco Cloud Services Router VM Guests
- Nexus 1000V and a VMWare DvSwitch, if no Nexus 1000V is available you can get away with just two DV-Switches :)
Basically, your going to have a topology that looks a little something like this:
Now if you haven't got the Cisco Cloud Services Router or you haven't set that up, which is really the first step you should check out the blog post from INE that explains how to configure it in VMWARE here.
OK, now that's set up. Let's quickly look at our topology.
So basically as you can see from the topology diagram our goal is to get the server 172.28.0.11 talking to 172.28.0.20 even though they are not in the same Layer 2 domain by using the CSR Routers to perform OTV for us :).
Let's quickly look at the initial configuration on the CSR Routers:
CSR 1:
interface GigabitEthernet1
ip address 172.27.0.1 255.255.255.252
end
CSR 2:
interface GigabitEthernet1
ip address 172.27.0.2 255.255.255.252
end
Can we ping across these hosts?
CSR1#ping 172.27.0.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.27.0.2, timeout is 2 seconds:
!!!!!
Yes we can, so now we get onto the next stage.
The next stage involves configuring our dvSwitch and our Nexus 1000V, if we are using the Nexus 1000V basically I create a port profile that trunks up the VLAN to my CSR:
port-profile type vethernet EXTEND_SITE_TRUNK
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 499-501
no shutdown
state enabled
VLAN 499 is my VLAN that the server will sit on, so I have created a port profile for that:
port-profile type vethernet EXTEND_VLAN_SERVER_1
vmware port-group
switchport access vlan 499
switchport mode access
no shutdown
state enabled
Now my server and my CSR #1 Router can see each other once I assign the port profiles to the appropriate VM's.
Ok now we have to duplicate the same sort of thing on the dvSwitch, because in this example we are trying to make the two hosts be on totally seperate switches that are not connected in any way. To do this I created a DVSwitch with No Uplinks and assigned from there:
As you can see I set the server to use the untagged port group and the Router to use the Trunk Port Group, the reasoning for this will become clear later.
Ok now I have all the basic infrastructure setup, let's get onto the OTV Config.
First thing I need to do on both CSR's is specify a site identifier and a site VLAN (Or in the case of CSR, which is not like configuring OTV on a nexus 7000, I specify a Bridge-domain)
CSR1(config)#otv site bridge-domain 1
CSR1(config-otv-site)#exit
CSR1(config)#otv site-identifier 0x2
And then the same on CSR2 but with a diffirent site Identifier
CSR2(config)#otv site bridge-domain 1
CSR2(config-otv-site)#exit
CSR2(config)#otv site-identifier 0x1
Now we need to configure the actual OTV Interfaces, the first thing to do is go to our join interfaces (which are the 172.27.0.0/30 links we setup previously) and specify IGMP Version 3 to support SSM multicast
CSR1(config)#interface GigabitEthernet1
CSR1(config-if)# ip igmp version 3
!
Now we repeat the above on CSR2 and then we are ready to configure the OTV Interface:
CSR 1:
interface Overlay1
no ip address
otv join-interface GigabitEthernet1
otv use-adjacency-server 172.27.0.1 unicast-only
otv adjacency-server unicast-only
!
CSR 2:
interface Overlay1
no ip address
otv join-interface GigabitEthernet1
otv use-adjacency-server 172.27.0.1 unicast-only
!
In the above example I have used an adjacency server that lives on CSR, by default OTV wants to use multicast but I had trouble getting this going with multicast, I believe the issue is related to the way I have the CSR's join interfaces connected to each other, so I did this to make life easier for myself for now.
As you can see one router (CSR1) is the adjacency server and I have configured CSR2 to point to this, you can have multiple adjacency servers specified for redundancy if you like.
OK now we need to configure the "extend" interface on our inside of our CSR, which is the interface on VLAN 499 on each switch respectively that heads down towards our end host servers.
Here is the configuration there:
interface GigabitEthernet2
no ip address
negotiation auto
service instance 2 ethernet
encapsulation dot1q 499
bridge-domain 200
!
end
I believe that the CSR has the same restriction that a Nexus 7000 has in terms of you can't have IP addresses assigned to interfaces that your trying to extend over the OTV. If anyone knows for sure if this is possible/not possible please post in the comments section :).
So the most important part of this configuration is where we specify a service instance, this will be how we map our OTV VLAN, as you can see the VLAN number is set here as well as a bridge domain.
If we now go back to our OTV interface:
interface Overlay1
no ip address
otv join-interface GigabitEthernet1
otv use-adjacency-server 172.27.0.1 unicast-only
service instance 2 ethernet
encapsulation dot1q 499
bridge-domain 200
no shutdown
!
end
Here we are actually mapping that service instance we created, now when we perform a no shut we should see the OTV tunnel come up, here are some useful troubleshooting commands to verify:
CSR1#show otv detail
Overlay Interface Overlay1
VPN name : None
VPN ID : 1
State : UP
AED Capable : Yes
Join interface(s) : GigabitEthernet1
Join IPv4 address : 172.27.0.1
Tunnel interface(s) : Tunnel0
Encapsulation format : GRE/IPv4
Site Bridge-Domain : 1
Capability : Unicast-only
Is Adjacency Server : Yes
Adj Server Configured : Yes
Prim/Sec Adj Svr(s) : 172.27.0.1
OTV instance(s) : 0
FHRP Filtering Enabled : Yes
ARP Suppression Enabled : Yes
ARP Cache Timeout : 600 seconds
This first command shows us that the AED has been elected (it's us), that our join interface is Gi1 and that our bridge ID is 1, also that we are using an adjacency server and that we ourselves are configured as an adjacency server. If you have errors in here you should revisit your config and make sure your not missing any of the critical commands such as the site bridge domain.
The next useful troubleshooting command is to establish if an ISIS adjacency exists between the hosts:
CSR1#show otv isis neighbors
Tag Overlay1:
System Id Type Interface IP Address State Holdtime Circuit Id
CSR2 L1 Ov1 172.27.0.2 UP 25 CSR1.01
If the OTV is successful the only state for this should be UP, you will also notice that you can see the hostname of your peer.
CSR1#show otv adjacency
Overlay 1 Adjacency Database
Hostname System-ID Dest Addr Up Time State
CSR2 001e.4925.fc00 172.27.0.2 00:20:11 UP
!
Here is another helpful command that just helps you verify the connectivity.
So far, it looks like everything is up and we should be good to go, let's do a ping from our host!
Success! Our OTV Tunnel is carrying the traffic, we can do some verification here too (this will also be helpful to you when troubleshooting if it does not work)
CSR1#show otv route
Codes: BD - Bridge-Domain, AD - Admin-Distance,
SI - Service Instance, * - Backup Route
OTV Unicast MAC Routing Table for Overlay1
Inst VLAN BD MAC Address AD Owner Next Hops(s)
----------------------------------------------------------
0 499 200 0050.5601.face 50 ISIS CSR2
0 499 200 0050.56a0.1aad 40 BD Eng Gi2:SI2
As you can see this shows some very useful information, you can see that the above routes are in the ISIS routing table, and what bridge-domain and VLAN they are attached to, also the owner helps you know if its local to you or over the OTV tunnel.
You can also see that it knows that one of the Mac's is actually pointed out interface Gi2:SI2 (Gigabit 2, Service Instance 2)
So this is quite a helpful command. Another useful command is:
CSR1#show otv arp-nd-cache
Overlay1 ARP/ND L3->L2 Address Mapping Cache
BD MAC Layer-3 Address Age (HH:MM:SS) Local/Remote
200 0050.5601.face 172.28.0.11 00:02:46 Remote
This will show you the ARP cache that the router will respond with whenever it see's an ARP request, this is part of the proxy-arp function that OTV performs to ensure that ARP flooding is not occuring too often, this would be a good place to look if you had recently changed IP address or recently changed a host on the OTV and where wondering why you couldn't see it, it could be that the ARP cache is out of date.
Finally, if your using the Nexus 1000V you can also use the following helpful troubleshooting command:
DCNexus1000V# show mac address-table vlan 499
VLAN MAC Address Type Age Port Mod
---------+-----------------+-------+---------+------------------------------+---
499 0050.56a0.115e static 0 Veth11 3
499 0050.56a0.1aad static 0 Veth2 3
499 001e.bd93.82bc dynamic 15 Veth11 3
499 0050.5601.face dynamic 248 Veth11 3
So Veth11 is the interface on my Nexus 1000V that goes towards my CSR router, so the fact that I have learnt the mac 0050.5601.face via this interface shows that the traffic is coming over the OTV.
I hope this helps someone out there, especially my fellow CCIE DC candidates.
Subscribe to:
Posts (Atom)
Popular old posts.
-
Hi Guys Having spent a lot of time with customers working on vPC deployments, I have found quite a few of the gotcha's for vPC that I w...
-
Hi Guys! This blog post is attempting to be the DEFINITIVE guide on Jumbo MTU, It's a topic that DOES MY HEAD IN! There are SO many ...
-
So some of the readers of this blog might already know this little trick, and what's more some of you might be surprised I didn't kn...