Tuesday, December 24, 2013

CCIE Rants Christmas Special: SSH Keys with IOS Routers

Hi Guys

Recently got myself a Mac Book Pro Retina, Most things I like, some things annoy me. The things I like are normally related to the underlying Unix in the OS, including SSH.

If your anything like me 99 percent of the time you log into routers etc you use a username and password, but Linux guru's (like my friend at www.sysadminblog.net)  have long been using SSH keys to create secure connections to there linux servers without having to enter a password every time.

We can do the exact same thing on IOS routers and it's relatively straightforward.

First, I will show you how to generate a ssh key in Mac OS X, I am sure that this process can be done with Putty, and in fact I think there is a putty program specifically geared for this exact reason called puttygen.

I will show the Mac-OS X commands, should be easy enough to figure it out for windows as i mentioned above.

OK, so first of all you need to generate the key, this is quite straightforward:

On your mac terminal, simply enter ssh-keygen and you can then accept most of the defaults, you will want to make sure you back up the private key somewhere (you can always regenerate the public key)

The Passphase is highly recommended: This means that if someone gets a copy of your private key they will still need this password, it also means that you will run a special bit of software called a "SSH Agent" that, when you first use the private key will prompt you for the pass phase to use it, it will only do this once per desktop login session (so if you shut your mac down or log out, next time you SSH in to a host your using SSH key authentication you will need to enter this key phase, but after that you won't have to enter that pass phase again. So pick something secure!

So, as per below enter the ssh-keygen command, choose a path to store the key, enter a pass phase.

Peters-MacBook-Pro:~ peterrevill$ ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key (/Users/peterrevill/.ssh/id_rsa)
Created directory '/Users/peterrevill/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/peterrevill/.ssh/id_rsa.
Your public key has been saved in /Users/peterrevill/.ssh/id_rsa.pub.
The key fingerprint is:

5b:11:d1:1d:2f:a6:2a:13:b3:09:6c:cd:f8:60:34:0c peterrevill@Peters-MacBook-Pro.local

OK next, cd to the directory and look at your new public key.

Peters-MacBook-Pro:.ssh peterrevill$ cd /Users/peterrevill/.ssh
Peters-MacBook-Pro:.ssh peterrevill$ cat id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrjoyJeKI/zqt446ntMihqzuHJV9F1Nh9pE+NiZS4khlWl2epnrxGvPIHEfF/P12OsCRMmHUggY1AmKxiQH/styVIGn+FC5y/GWlPwZe7ZDPMUDeoJJCOP5TZWQoIBnF2S+07rKTxlBxtm2BTlJzx+IteQHwA+G/hWLtQL24nWO3/g/urrttD5mTgncHPtTNkc4CiVrr3Av64eO3QlU8ZuGYYFBTrL4nuYHIIywtHT91UAFokB/oddk7vFwwbIpjddkmAvnkUlxHR0719CfxAXjFRVQmCqiVy9ewosUhDZiBXm/NJWvi0CXl/ZVWlsYAqg0CpZ4s5u+V5E0R6/MnKv peterrevill@Peters-MacBook-Pro.local

We need to paste the output of our cat command into the router, so at this point it's time to login to the router. It should be obvious you need SSH enabled, and you will want SSH version 2.

Now, the thing about this process that I have found, is that if you have a user who already has a password, even if your using SSH keys it prompts you for the password, there may be a way to resolve this but not that your humble friend (Me!) could find. So if anyone does know a way to stop this behaviour and allow people to login with EITHER SSH key OR password but not both please post it in the comments :). 

So, to account for this we make a new user and we don't assign them a password, we do assign them privilege 15 though so that the enable password does not have to be entered. Once you have entered the final command key-string come back to this page before continuing.

Issue the following commands:

HomeRouter2921(config)#username test privilege 15
HomeRouter2921(config)#ip ssh pubkey-chain 
HomeRouter2921(conf-ssh-pubkey)#username test

At this point your ready to enter the key, now unfortunately IOS has a limit per line of 250 characters, so you will need to copy 250 characters at a time and go from one line to the next, being careful not to miss any characters whatsoever. (Why they can't make this like the certificate enrolment process where you paste everything in in one go i have no idea!)

It will probably look a little something like this:


The start of the key will show: ssh-rsa, you have to copy this in and copy the plaintext bit at the end as this is the description.

Once this is done, you are ready to test it out! That easy hey?

Go to a terminal in Mac OS X and issue the ssh command, we will put the -v flag this time so you can see the process but afterwards you can do away with the -v and just use ssh test@ 

If you assigned a pass-phase to your private key (which you should have done), you will now be prompted by Mac-OS-X itself to enter the pass phase for the private key, after that you should not have to enter this pass phase again until turn off your mac, even subsequent SSH sessions with OTHER routers will be fine.

Peters-MacBook-Pro:.ssh peterrevill$ ssh -v test@
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: Connecting to [] port 22.
debug1: Connection established.
debug1: identity file /Users/peterrevill/.ssh/id_rsa type 1
debug1: identity file /Users/peterrevill/.ssh/id_rsa-cert type -1
debug1: identity file /Users/peterrevill/.ssh/id_dsa type -1
debug1: identity file /Users/peterrevill/.ssh/id_dsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version Cisco-1.25
debug1: no match: Cisco-1.25
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-md5 none
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024 p="" sent="">
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 32:a3:4b:36:88:94:fc:c0:b4:17:06:b1:21:c4:c1:2f
debug1: Host '' is known and matches the RSA host key.
debug1: Found key in /Users/peterrevill/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_ACCEPT received

The above output shows you that we sent a key to the router and it accepted it, and then we get on with the rest of the process...

debug1: Authentications that can continue: publickey,keyboard-interactive,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/peterrevill/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (public key).

Authenticated to ([]:22).

We have successfully authenticated!


We are now logged into the router and ready to start configuring!

That about wraps it up for me until the end of the year I believe, and I also reckon i won't be doing any blog posts until some time in February as I am about to go on my Holiday/Snowboarding Trip to Japan! If your going to be in Myoko Kogen between 11th of Jan to 15th of Feb or you will be at Hakuba for some of Feb Send me an email! :) Peter@peterrevill.com

I want to thank everyone for there continued support, my friends and family and everyone who has been really positive about this blog, I have found it very rewarding to be able to help people and quite a few people whom i have not met have told me hey I know you i read your blog. Really nice Feeling!

Enjoy your Christmas Holidays!

Wednesday, December 11, 2013

Anyconnect SSLVPN

Finally! I found a definitive guide on how to install a 3rd party cert easily:

Saturday, December 7, 2013

UCS emulator troubleshooting tips

Hi Guys!

Found a great site with some UCS emulator troubleshoooting tips that you might find helpful!


Tuesday, November 26, 2013

802.11 AC and the Cisco AC Module for 3600 Access Points

Hi Guys

This blog post is about the Cisco 3600 access point, the 802.11 AC Module (wave 1) available for that access point and finally the software called "chanalyzer" available to help you with detecting interference and leveraging your Cisco Cleanair access points.

So Incase you where not aware, Cisco have had some access points out now for quite some time called the Cisco 3600 Access Points:

The picture above shows them in all there glory, along with the 802.11 AC Module.

The 802.11 AC module is I believe, an industry first in that your existing 802.11N access points (the 3600's) can be field upgraded to 802.11AC.

The module simply plugs into the back of the AP, as shown in the following pictures:

As you can see, it's quite straight forward! Simply place the module into the slot shown on the AP and then tighten the finger-tight screws

- If 802.11 Ac can offer "gigabit wifi", and its maximum theoretical speed is 1.3 Gigabits per second, don't I need more than 1 gigabit connection from the AP to the switch?

No! 1.3 Gigabit is the maximum theoretical PHYSICAL speed of the radio media, the actual throughput onto the LAN will be less, quite a bit less as we will see.

- Does the 802.11 AC Module require more power?

Yes, you need to make sure your switch supports POE+, the output below shows the wattage required for a 1142 vs the 3602 with the 802.11 AC Module:

homeSwitch3750#show power inline

Module   Available     Used     Remaining
          (Watts)     (Watts)    (Watts)
------   ---------   --------   ---------
1           480.0       35.4       444.6
Interface Admin  Oper       Power   Device              Class Max
--------- ------ ---------- ------- ------------------- ----- ----
Gi1/0/5   auto   on         20.0    AIR-CAP3602I-N-K9   4     30.0
Gi1/0/18  auto   on         15.4    AIR-LAP1142N-N-K9   3     30.0

- Is it really just plug and play?

Yes! Simply turn the AP off, plug the module in, turn it back on and make sure your running WLC Code of version 7.5 or later, then enable 802.11 AC in the controller.

What kind of speed can we really get from 802.11AC?

I tested with the following equipment:

ASUS PCE-66 802.11 AC Wireless Card
Windows 7
Cisco 3600 Access Point
 Laptop connected to gigabit network.

The enviroment was a home setup with quite a bit of wireless interference.

The first step was to establish a gigabit ethernet (wired) benchmark, which resulted in:

722 Megabits per second

The following tests where undertaken:

802.11AC (5.0ghz only, 80 Mhz)
5.0Ghz, TCP, Close Average Speed: 166 Mbits/Sec
5.0Ghz, TCP, 1 Room away Speed:  102 Mbits/Sec

As you can see, it's not even in the same ballpark as what we can get for wired, but it's a significant improvement over 802.11N:

802.11n (5.0Ghz. 40MHZ)
5.0 GHZ TCP Close average speed: 108 Mbits per Second
5.0 GHZ TCP, 1 Room Away Speed: 69.3 Mbits Per Second

802.11n (2.4 GHZ)
2.4 GHZ TCP, Close Average Speed: 71.1 Mbits Per Second
2.4 GHZ, TCP, 1 Room Away Speed:, 50.1 Mbits Per Second

The results above show that 802.11 AC could potentially be a worthwhile improvement, but is certainly not the gigabit wifi we have been led to believe it could be.

The speeds, as rough as they are closely match those achieved by Tom's Hardware:


I discovered Chanalyzer at a customers site and wanted to share it with you guys, it' s an exceptionally useful tool when combined with Cisco Clean Air


The software can be used to connect to a Cisco Access Point running clean air to retrieve spectrum information in real-time!

The above screen looks quite busy but is essentially showing interfering  devices, in this case an xbox controller. This real-time analysis can really help you realize what might be affecting the network.

After dowloading Chanalyzer, you will need to connect to the AP:

The NSI key stands for "network spectrum interface key" and can be found on the WLC Access point page:

This can be very useful for troubleshooting Clean Air Problems.

I hope you enjoyed this blog!

Incidentally, this is my first blog post as a Cisco Data Center Champion, an honor I was recently bestowed by Cisco. Hopefully I can live up to the name!

Thursday, November 7, 2013

isdn sending-complete

This command prevents h.323 dialling delays

Quick blog post guys just so i remmeber myself

Monday, October 28, 2013

Quick and easy way to get around the CIsco UCS - Login Error: java.io.IOException: Invalid Http response Login Error: java.io.IOException: Server returned HTTP response code: 400 for URL: http://x.x.x.x:443/nuova

Hey Guys!

You might have experienced the exceptionally annoying error with Cisco UCS and the latest update to Java (and apparently the fault is a java programming fault not Cisco UCS) where after trying to login to Cisco UCS you get the following error:

Login Error: java.io.IOException: Invalid Http response
Login Error: java.io.IOException: Server returned HTTP response code: 400 for URL: http://x.x.x.x:443/nuova

Here is a quick and easy work-around to enable HTTP via the SSH command shell, you can then HTTP into the device instead of HTTPS and at least that will get you on there quickly instead of having to stuff around with Java,

naturally I would recommend disabling HTTP since everything is transmitted in clear text once the issue is resolved

UCS# scope system 
UCS /system # scope services 
UCS /system/services # enable http 
UCS /system/services # disable http-redirect 
Warning: When committed, this closes all the web sessions. 
UCS /system/services* # commit

Monday, October 7, 2013

CCIE DC: How I passed the CCIE DC

Hi Guys!

As per the subject i passed my CCIE DC and in this blog post I will describe the process that worked for me. What worked for me might not work for you! So keep that in mind.

OK I am going to assume in this post you have already passed the CCIE DC Written and are looking at the lab.

There are several aspects to the CCIE DC Lab that are worth considering, the first is: Lab Rack availability, at the moment at least, is quite sparse, until this situation improves, I recommend at the beginning of your studies for the lab (and assuming you studied hard for the written) I would say make a lab booking within 4 to 6 months from when your ready to knuckle down and start studying hard.

The other aspect is Rack Rental availability from Vendors such as INE and IPExpert is also quite tight at the moment., which makes things difficult, in this blog post we will go through some of the stratergys to help with that situation.

Before I start getting too much into depth, let me take a quick chance to address the "Big two" of vendor training, INE and IP Expert.

I am absolutely not a fanboy by any strech of the imagination: I will use products from either vendor to achieve my goal. I just want to pass, each company has pro's and con's and by combining both, your more likely to get a nice mix. Don't dismiss other options such as traditional training from Cisco Vendor Partners like Firefly, Fastlane etc. It was through a firefly Course that I met a very good friend of Mine Adriaan Steyn,Believe me when i tell you he taught me PLENTY of great info about the Nexus line of switches, I also did a UCS Course with Housley Communications and that too was great training.

Good training, like good products is not restricted to a certain vendor: You might love Sony Headphones but hate Sony TV's for example, don't restrict yourself to a "Oh my training partner is XYZ", That is dumb.  Combine all the training you can, learn from whoever you can.

(Full Disclosure: I am likely to be doing training material for a training vendor at some point in the future.)

OK let's start talking about the study method I used.

First of all, You need to learn the protocols and topics on the blueprint, that much should be obvious, to study, i hand write all my notes, i always hand write my notes, never type: your brain is wired through thousands of years of evolution to hand write, you absorb the information more readily than typing, it is too easy when typing notes to just type and not really think about what it is your typing (especially if like me you can touch type). Hand writing forces you to slow down and the fact that you have to write it, means invariably you will reword it to make it shorter to write, meaning you will understand it better. Plus it's much much easier to draw a network engineers favorite tool: a diagram.

So, I got a book, I printed a copy of the blueprint out and stuck it on the inside of the book and started watching the INE video's, I went through every single one, hand writing notes as I went along. I started with the topic that most interested me and was newest to me, storage. (Incidentally, many idiots will tell you that storage is not a big part of the CCIE DC: They are wrong, it's a huge part, so many things depend on storage that if you don't know it backwards you will not pass)

Starting on a new topic got me enthusiastic and gave me interest I needed to stay motivated.

I then bought myself two Cisco MDS 9216i Switches, I cannot recommend this enough, you SHOULD buy these switches for your own home lab, they are about 500 bucks on ebay (I am selling mine if anyone is interseted, contact me peter@peterrevill.com), I am honestly not just pimping them because I am selling mine: it's INCREDIBLY useful to be able to test the storage stuff in a limited fashion in your own home, and for such a low price this is one of the few things in CCIE DC you can play with yourself. Be warned: they are VERY noisy. Make sure you get ones with Generation 2 modules, Generation 1 modules CANNOT do certain types of port channels and it can become very frustarting.

So now I was watching the INE Storage videos, taking notes, then testing on my home lab equipment what I could, certain topics like FCoE I couldn't test so at that point I would do rack rentals from either IPexpert or INE: Both of them have there pro's and con's, with IPExpert you get access to a FULL lab of the ENTIRE CCIE DC, _but_ the availability is quite low at the moment, with INE (and this may have changed, you would be best off asking them), you don't have access to the whole N7k yourself so you can't make your own VDC's for example, and you can't create a storage VDC, I also _BELIEVE_ that the current N5k is a 5010 or 5020 and not a 5548 which also limits some of the things you can do. You can still do a HECK of a lot with the 5020's, but if your desperate to make sure you have the exact same equipment as the DC Lab then you will want to get yourself some IPExpert time.

Speaking of the rack rental systems, I must say; the INE rack rental system that you use for booking and organizing etc is _EXCELLENT_. I did make a humble solution for improvement on it but all in all it is excellent, you can tell they have spent a lot of time working on it and it shows.

So now I am busy going through this topic and at this point I have not done any workbooks or labs, just going through the theory, testing it out myself. I started posting on my blog as well as this was going on. I strongly recommend starting a blog: being able to successfully communicate a concept to someone else means that you UNDERSTAND that concept.  Plus you can meet some great people through your blog, and it's a great way to organise your own thoughts and opinions.

So I kept this process up through all the individual topics, UCS, Virtualization, Ace, pretty much all the video's provided by INE I went through and studied each of those topics. Once this was finally done, it was time to start the practice labs.

Each vendor has technology labs and full scale mock labs and both have as I mentioned Pro's and Con's, I can't recommend enough doing both. You will always learn more. Data Centre, being a new topic has a few people with somewhat of a misunderstanding of some topics, this is probably the first CCIE I have ever done where I have _NOT_ been able to take whatever is said in the workbook and solution guide as gospel: the technologies are just too new, for some topics you really do have to study yourself, if the answer is unclear go ahead and check it. Be careful listening to other people, many people out there post flat out wrong answer to questions, the best way and only way in my opinion is to _TEST IT YOURSELF_

Perfect example, During my bootcamp there was a discussion about vPC and what kind of BPDU's vPC would send, which switch (primary and secondary) would send BPDU's and what commands you could do to tweak it, so instead of everyone arguing and trying to show how big there brain was, we said let's do a debug spanning-tree bpdu event or whatever the command is and CHECK.


There is SO much misinformation out there that you just HAVE to check for YOURSELF what works and what does not, how the technology works, this is how i try and structure my blog: I theorize how I believe a protocol should work or something should work, then I test my hypothesis, and I humbly submit my findings to you, my blog reader. It's a little thing you might have heard of called the scientific method: Adapt this in your studies, trust nothing, verify it yourself, prove to yourself you have an understanding as to exactly how it works.

To compliment my study regime, I started reading quite a few books, my friend Ron Fuller and David Yansen have a great Cisco Nexus book, for example, in no particular order I read the following books pretty much from cover to cover.

(Full Disclosure: If you buy the books below using the link on my webpage here I get a small percentage of that sale, it does not add any cost to you for the price of the book, I would consider it a favour if you buy the book to please use my links below.)

The first is a Nexus topic book, this was a great great book. There are some very good topics covered very well here and it really complimented the INE videos and helped me fill in the gaps of my knowledge. If you only buy one book for the CCIE DC studies make it this one.

NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures (2nd Edition) (Networking Technology)
The second book is a great book on Cisco UCS, It's a little dated at the moment but I believe a second edition is on it's way
Cisco Unified Computing System (UCS) (Data Center): A Complete Reference Guide to the Cisco Data Center Virtualization Server Architecture (Networking Technology)
This book Is also highly recommended in the CCIE DC community, I must confess I have not got a copy myself but everyone who has read it raves about it:
Data Center Virtualization Fundamentals: Understanding Techniques and Designs for Highly Efficient Data Centers with Cisco Nexus, UCS, MDS, and Beyond
The final book I will recommend is a multicast concept book, Multicast is an important part of the CCIE DC and this book is extremely well written, actually kind of funny to be honest, the author is a funny guy and cracks jokes the whole way through, multicast was a mystery to me until i read this book for my CCIE R&S, it is now a tiny bit dated but multicast has not changed all that much so was a great resource for me. Developing IP Multicast Networks, Volume I (paperback)
At this point i was ready to begin discussing my thoughts on protocols with my friends and collegues studying for the same exam (or performing Data-Centre related work out in the field.) This is a crucial step: your friends will challenge your knowledge, if you don't know a topic well and they do, they will unveil this weakness to you: My friends and I did this to great success, we would give each other hairy questions and make friendly wagers on how a particular technology is meant to work, like how does vPC peer-switch actually work.  I confess to losing quite a few of these wagers!

I became active in the CCIE DC Facebook Group which you should definately join if your on this quest, you will get some great questions and tips and tricks for all your Data Centre work in this great little community. Respect the rules and join! (Full Disclosure: I am an admin of this facebook group, but I receive no compensation)

I also became active on twitter and tried my best to help people out there: Good questions can help you understand a technology better, but be careful not to claim you have an answer to a problem if you don't. Try not to mislead people! Too much misinformation out there.

For rack time, A great tool is the Cisco PEC: Use it! Remember, even though they have particular labs, if the equipment matches what your trying to learn your NOT restricted to JUST doing the labs they list in the PEC, the PEC is a valuable resource: use it.

So after I finished up the technology labs and a few mock labs I went on my CCIE Data Centre Bootcamp with Ipexpert, I always do this as my final preperation of study: The bootcamps should be your final step, don't go to a bootcamp as one of your first steps, it will be wasted on you. You can't possibly cram the massive amount of information you need to know and massive topic list into just 2 weeks or a week or however long you have, your bootcamp should be about solidifying everything and making sure you know what's going on. A sign your on the right track is throwing out great tricky questions to your lecturer.

After this I was ready for my first attempt and took it in brussels, while I was quite close, I failed, I was absolutely distraught: I really thought I had nailed it, I finished quite early, my mistake was not reading the questions carefully. I was humbled by the first attempt, it showed me that although I might have most of the technologies understood and I definitely have the speed i did NOT have the careful reading that is required of this CCIE DC exam.

I quickly booked a second attempt and during the time between first attempt and second I just studied my notes again, kept everything fresh as I already had the knowledge I just didn't have the careful reading required of the exam. I managed to pass the second time after being careful.

I hope this helps someone out there, to all my loyal readers and people with kind words: it honestly makes me SMILE from EAR TO EAR when I know that one of my blog posts helped you! I like to help people it's in my nature, I wish you all the luck in the world in your quest to obtain CCIE DC.

IP Expert running a free ipad competition

Hi Guys!

IPExpert are running a competition to win a free ipad. check it out below.

If you’re interested in winning a free iPad mini? Just follow us on Twitter or “Like” our Facebook page and be entered into a weekly drawing! A winner will be announced every Friday when we post our success blog!

CCIE DC: Definitive Jumbo frames

Hi Guys!

This blog post is attempting to be the DEFINITIVE guide on Jumbo MTU, It's a topic that DOES MY HEAD IN!

There are SO many possible iterations, there's

MTU with Nexus 5k
MTU with Catalyst Switches
MTU with MDS
MTU with Nexus 7k
MTU on SVI Interfaces
MTU on physical interfaces
MTU on UCS (Both on the FI itself and the vNIC'S)
MTU on C-Series
MTU on Nexus 1000v

What interaction does MTU have on VPC, Fabric Path, Port Channels, OTV? Routing Protocols?

What interactions does MTU have with FCoE? FC? SAN-Port-Channels?

MTU on FC?
Why 9216 vs 9000?

MTU on Nexus 7000

So this is the most complicated for me at least of the jumbo MTU discussion, let's start at the beginning:

A Nexus 7000 by default, has the following command configured:
system jumbomtu 9216

Under a VDC, I could not remove this command:

N7K5(config)# show run all | inc jumbo
system jumbomtu 9216
N7K5(config)# no system jumbomtu 9216N7K5(config)# show run all | inc jumbo
system jumbomtu 9216N7K5(config)# show run all | inc jumbo
system jumbomtu 9216

I am not sure if this is a limitation of VDC's or simply something I am doing wrong, but regardless I was unable to turn off this command.

This fits with the story I have always heard, that a Nexus 7000 is enabled for jumbo frames, BY DEFAULT. And that you do not have to do ANYTHING, but of course it's a little bit more complicated than that.

If you look at the interfaces you have with a show run interface command, you will see an interesting default:

#show run all | beg 1/17

interface Ethernet1/17
 (Output omitted)
  mtu 1500
  snmp trap link-status
  logging event port link-status default
  logging event port trunk-status default
  bandwidth 10000000

 (Output omitted)

The MTU command is on ALL interfaces when you do a show run all, what the hell is it? Does it over-write my system Jumbo MTU? Do I have to set it too?

If you look at an interface with show int, you annoyingly see exactly the same thing:

N7K5(config-if)# show int eth1/17 | inc MTU
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

Yet we are lead to believe by certain blog posts that because we have the system jumbo MTU command we don't have to do anything.

Let's try not changing the value, and doing a ping, and seeing what happens

N7K6# ping df-bit packet-size 8000
PING ( 8000 data bytes
Request 0 timed out
--- ping statistics ---
2 packets transmitted, 0 packets received, 100.00% packet loss
N7K6# ping df-bit packet-size 1500
PING ( 1500 data bytes
Request 0 timed out
--- ping statistics ---
2 packets transmitted, 0 packets received, 100.00% packet loss
N7K6# ping df-bit packet-size 1472
PING ( 1472 data bytes
1480 bytes from icmp_seq=0 ttl=254 time=1.273 ms
1480 bytes from icmp_seq=1 ttl=254 time=0.881 ms
1480 bytes from icmp_seq=2 ttl=254 time=1.182 ms
1480 bytes from icmp_seq=3 ttl=254 time=1.179 ms
1480 bytes from icmp_seq=4 ttl=254 time=1.185 ms

--- ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.881/1.139/1.273 ms

No jumbo frames for us! Maximum we could do is 1472.

Let's show that we just have the default config:

N7K6# show run int eth1/17

!Command: show running-config interface Ethernet1/17
!Time: Sun Jun 30 05:50:14 2013

version 6.0(2)

interface Ethernet1/17
  switchport mode trunk
  no shutdown

N7K6# show int e1/17 | inc MTU
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

So! Right now we can see clear as day: if we don't change the MTU value, it does not appear to work!

So let's try changing this value and see what happens

N7K5(config-if)# int eth1/17
N7K5(config-if)# mtu 9216

Let's see if we change it to 9216 if our output has changed:

N7K5(config-if)# show int eth1/17 | inc MTU
  MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec


It has changed the output, do our pings work now?

N7K6(config-if)# exit
N7K6(config)# exit
N7K6# ping df-bit packet-size 1472
PING ( 1472 data bytes
1480 bytes from icmp_seq=0 ttl=254 time=1.457 ms
1480 bytes from icmp_seq=1 ttl=254 time=0.86 ms

--- ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.77/0.952/1.457 ms
N7K6# ping df-bit packet-size 1500
PING ( 1500 data bytes
1508 bytes from icmp_seq=0 ttl=254 time=1.341 ms
1508 bytes from icmp_seq=1 ttl=254 time=1.067 ms

--- ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 1.067/1.235/1.343 ms
N7K6# ping df-bit packet-size 8972
PING ( 8972 data bytes
8980 bytes from icmp_seq=0 ttl=254 time=1.872 ms
8980 bytes from icmp_seq=1 ttl=254 time=6.154 ms
Request 2 timed out

So the RESULTS ARE IN!  Despite all the "Experts" on the internet claiming otherwise, who lead to me being lead astray myself, the Nexus 7000 system jumbo mtu command is NOT enough.  on M interface cards, you _MUST_ set the MTU

Regardless of L2 VS L3 Interface

Here is absolute proof:


I tested it myself, see below

N7K5(config-if)# mtu 1600
ERROR: Ethernet1/17: MTU on L2 interfaces can only be set to default or system-jumboMTU

You can only set the MTU to either your configured jumbo MTU size, or the normal MTU size on L2 interfaces, l3 interfaces you can configure what you want.

N7K8(config-if)# no switchport
N7K8(config-if)# mtu 4444


 Note you can change the MTU individually on both the M line cards OR F Line cards, but your better off with M line cards using the system-QOS, as we will see below

  What about a system-QOS Class?

Guess what, all the above changes when it comes to F-Based Linecards. Although you can manually set the F line card MTU just like you can on the M line cards and it will work, you can change it globally using network-qos:


F-Based line cards require you to change the System-QOS Class just like on 5k, but unlike a 5k, on a 7k F1 line-card  if you change it, the value is changed under the interface:

SW1-1(config)# show int e4/1

Ethernet4/1 is up

  Dedicated Interface

  Hardware: 1000/10000 Ethernet, address: c464.1348.b2d8 (bia c464.1348.b2d8)

  MTU bytes (CoS values):  MTU  9216(0-2,4-7) bytes  MTU  2112(3) bytes

  BW 10000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255

Here is the config that was applied to make this happen:

policy-map type network-qos default-nq-7e-CCIE

  class type network-qos c-nq-7e-drop

    congestion-control tail-drop

    mtu 9216

  class type network-qos c-nq-7e-ndrop-fcoe


    mtu 2112


 So you can see that it matches up. So it makes it quite a bit easier to see what is going on! Great!

Here is a handy command on the Nexus 7k to see what is going on too, in the below example, I have an N7k which does NOT have any QoS config, because I was using a rack that you only have access to a single VDC on.

N7K8# show system internal qos network-qos hw-config module 2 | inc MTU
MTU         = 1500 [FCoE: No] - This line shows the MTU Value for CoS 0 on this hardware
MTU         = 1500 [FCoE: No] - This is MTU value for Cos 1
MTU         = 1500 [FCoE: No] - Cos 2
MTU         = 1500 [FCoE: No] - Cos 3
MTU         = 1500 [FCoE: No] - Cos 4
MTU         = 1500 [FCoE: No] - Cos 5
MTU         = 1500 [FCoE: No] - Cos 6
MTU         = 1500 [FCoE: No] - Cos 7
Interface    Config  Oper(VLs) MTU (value)

MTU on Nexus 5000

The Nexus 5000 is probably the simplest of all in terms of what we can and can't do with Jumbo MTU's.

The thing about the Nexus 5k is that it shares a similar architecture with the Nexus 7000 F1 Line cards, so thus it does not suprise me, that for Nexus 5K the only thing you have to do, to enable jumbo frames, is change the system QOS Class

policy-map type network-qos jumboMTU5k
  class type network-qos class-fcoe
    pause no-drop
    mtu 2158
  class type network-qos class-default
    mtu 9216

Just like on the Nexus 7000, if your working with an L3 interface, you must set the MTU manually

The annoying thing about the configuration on nexus 5000, is that the interface will still show the MTU as 1500:

N5K5(config-if)# show int eth1/6 | inc MTU
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec

This is a known bug:

CSCsl21529, Symptom: An incorrect MTU value is displayed in the show interface command output. The Cisco Nexus 5000 Series switch only supports class-based MTU. Per-interface level MTU configuration is not supported. The switch supports jumbo frames by default. However, the show interface command output currently displays an incorrect MTU value of 1500 bytes. 
But if you check the queuing on the interface, which is the QoS Applied as part of the global policy, it will show the correct MTU:

N5K5(config-if)# show queuing interface eth1/6 | inc MTU
    q-size: 243200, HW MTU: 9280 (9216 configured)

So let's do a ping between our two Nexus 5k's enabled for this.

N5K8# ping df-bit packet-size 9000
PING ( 9000 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

--- ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss
N5K8# ping df-bit packet-size 2000
PING ( 2000 data bytes
2008 bytes from icmp_seq=0 ttl=254 time=1.523 ms

Hmmm.. I can do a certain size but not over a certain size, this could simply be some sort of control-plane policing on the nexus 5k, as if i ping THROUGH these devices, on my nexus 7k's, I can do up to 9000 bytes.

So long story short, Nexus 5k: change the system class, your done. Change the MTU on any L3 interfaces you want to use. Done.


There is no Trick to this aspect, it is as simple as this: M1, SVI, Nexus 5k, Nexus 7k, none of that matters: You must set an MTU for layer 3 interfaces in order for those interfaces to support jumbo frames. Think of it like setting your operating system MTU

-- Subtopic: MTU interaction with routing protocols

I am not going to spend too much time on this, you should know if your doing CCIE DC that many routing protocols require you to match your MTU 

MTU on Port-Channels

On Nexus 7k, As you would expect if you add ports to a port-channel, the port-channel will inherit the MTU of the physical ports, or you can just change the MTU on the port-channel, either way you must configure the MTU on the port-channel even if it is layer 2 or layer 3.

On Nexus 5k, as we expect, the only thing that matters is System Class. As long as that allows jumbo MTU you are laughing.


With VPC on 7k the peer-link is always set to a Jumbo MTU, you cannot change this:

N7K7(config-if)# mtu 1500
ERROR: port-channel10: Cannot configure port MTU on Peer-Link.

There is even a bug:
From 4.2 NX-OS release onwards, VPC peer-link MTU is set to 9216 + padding.
ISSU SW upgrade form 4.1 to 4.2 will keep VPC peer-link MTU settings to prior 4.2 change and shut/no shut is needed for changes to take effect.

if you have a port-channel configured and are using it with vPC, on the Nexus 7k you must enable the MTU to be jumbo under the port-channel on both interfaces.

To test this, I configured a back to back vPC between a Nexus 5k set and a Nexus 7k set.

Here is the config:


 N5K7# show run | sect vpc|feature|port
feature lacp
feature vpc
feature lldp
vpc domain 2
  peer-keepalive destination
interface port-channel10
  switchport mode trunk
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
interface port-channel11
  speed 10000
  vpc 11


 N5K8# show run | sect vpc|feature|port
feature lacp
feature vpc
feature lldp
vpc domain 2
  peer-keepalive destination

interface port-channel10
  switchport mode trunk
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
interface port-channel11
  speed 10000
  vpc 11

N7K7# show run | sect vpc|feature|port-
feature interface-vlan
feature lacp
feature vpc
vpc domain 1
  peer-keepalive destination source vrf default
interface port-channel10
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
interface port-channel11
  vpc 11


N7K8# show run | sect vpc|feature|port-
feature interface-vlan
feature lacp
feature vpc
vpc domain 1
  peer-keepalive destination source vrf default
interface port-channel10
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link
interface port-channel11
  vpc 11


In my above example, the two vPC peers could ping each other with jumbo frames (i.e. N7k1 could ping N7k2 with jumbo frames and N5k1 and N5k2 could ping each other with jumbo frames) BUT the 7k could not ping a 5k with jumbo frames, this is because the port-channel 11 (which is the one that connects the 5k and the 7k), was not configured for jumbo frames on the N7k side, it IS configured for jumbo on the 5k, because the 5k only uses the system class remember? so we need to set the 7k port-channel 11 to have jumbo mtu:

N7K8(config)# int po11
N7K8(config-if)# mtu 9216

(Done on BOTH N7K's)

Now let's check it out

N5K7# ping df-bit packet-size 2000
PING ( 2000 data bytes
2008 bytes from icmp_seq=0 ttl=254 time=1.418 ms
2008 bytes from icmp_seq=1 ttl=254 time=1.081 ms
2008 bytes from icmp_seq=2 ttl=254 time=1.086 ms
2008 bytes from icmp_seq=3 ttl=254 time=0.971 ms
2008 bytes from icmp_seq=4 ttl=254 time=1.09 ms

Done, all very logical, makes sense that a vPC port-channel would act just like any other port-channel, the only real take-away from this is that the peer-link always has an MTU set to jumbo and you can't change it.

MTU and FabricPath

I have covered fabricpath MTU in a previous topic but will cover it here for the sake of completeness. FabricPath uses an outer header that makes it slightly bigger than the normal ethernet header, however because it's actually NOT an ethernet frame and instead is a Fabricpath frame with an ethernet header encapsulated inside, if your linecards (F Line cards and Nexus 5000's) support it, this overhead does not need to be taken into account.

However, if you have a VLAN that you want to run Jumbo frames on and this VLAN is also fabricpath enabled, you need to specify the interfaces as jumbo which we will cover here.

Here is our basic FabricPath Config, super simple and easy:

install feature-set fabricpath
feature-set fabricpath
vlan 10

  mode fabricpath
int eth1/9 - 10
  switchport mode fabricpath
  switchport mode fabricpath
fabricpath domain default

We then configure vlan 10 interface so we can ping between the hosts.

Look at our first bit of output that is relevant:

N5K-p1-2# show fabricpath isis interface  brief
Fabricpath IS-IS domain: default
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
Ethernet1/9  P2P   1     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/10 P2P   2     Up/Ready   0x01/L1   1500 40      64          1/1

So right now our fabric path is only between two 5k's and as you can see the MTU is set to 1500.

Let's test with a ping.

N5K-p1-2(config-if)# show run int vlan 10
interface Vlan10
  no shutdown
  mtu 9216
  ip address

N5K-p1-1# ping df-bit packet-size 1500
PING ( 1500 data bytes
Request 0 timed out
Request 1 timed out

No dice, let's try when we modify the default QoS Policy on the Nexus 5000.

N5K-p1-1(config-sys-qos)# show run | sect policy-map
policy-map type network-qos JUMBO
  class type network-qos class-default
    mtu 9216

N5K-p1-2(config-pmap-nq-c)# system qos
N5K-p1-2(config-sys-qos)# service-policy type network-qos JUMBO

Let's try a ping again:

N5K-p1-1# ping df-bit packet-size 8972
PING ( 8972 data bytes
8980 bytes from icmp_seq=0 ttl=254 time=3.328 ms
8980 bytes from icmp_seq=1 ttl=254 time=4.898 ms
8980 bytes from icmp_seq=2 ttl=254 time=4.958 ms
8980 bytes from icmp_seq=3 ttl=254 time=19.405 ms
8980 bytes from icmp_seq=4 ttl=254 time=3.161 ms

Jackpot, but note the show fabricpath isis interface output does not change:

N5K-p1-1# show fabricpath isis inter brief
Fabricpath IS-IS domain: default
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
Ethernet1/9  P2P   1     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/10 P2P   2     Up/Ready   0x01/L1   1500 40      64          1/1

We have kind of come to expect that though on the 5K so no worries.

Let's involve the 7k.

N7K-1-2# ping df-bit packet-size 1500
PING ( 1500 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out

--- ping statistics ---
4 packets transmitted, 0 packets received, 100.00% packet loss
N7K-1-2# show fabricpath isis interf brief
Fabricpath IS-IS domain: default
Interface    Type  Idx State        Circuit   MTU  Metric  Priority  Adjs/AdjsUp
Ethernet1/1  P2P   1     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/2  P2P   2     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/3  P2P   3     Up/Ready   0x01/L1   1500 40      64          1/1
Ethernet1/4  P2P   4     Up/Ready   0x01/L1   1500 40      64          1/1

No dice until we up the interface MTU manually (or change the system QOS globally)

N7K-1-2(config)# int eth1/1 - 8
N7K-1-2(config-if-range)# mtu 9216
N7K-1-2(config-if-range)# exit
N7K-1-2(config)# exit
N7K-1-2# ping df-bit packet-size 1500
PING ( 1500 data bytes
1508 bytes from icmp_seq=0 ttl=254 time=10.147 ms
1508 bytes from icmp_seq=1 ttl=254 time=3.067 ms
1508 bytes from icmp_seq=2 ttl=254 time=0.804 ms
1508 bytes from icmp_seq=3 ttl=254 time=0.833 ms
1508 bytes from icmp_seq=4 ttl=254 time=2.296 ms

--- ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.804/3.429/10.147 ms
N7K-1-2# ping df-bit packet-size 8972
PING ( 8972 data bytes
8980 bytes from icmp_seq=0 ttl=254 time=1.652 ms
8980 bytes from icmp_seq=1 ttl=254 time=6.884 ms
8980 bytes from icmp_seq=2 ttl=254 time=6.992 ms
8980 bytes from icmp_seq=3 ttl=254 time=7.053 ms
8980 bytes from icmp_seq=4 ttl=254 time=15.796 ms

Done and done.


To test this one we used the most simple of configs to show you how it is done.

interface Overlay1
  otv join-interface Ethernet1/25
  otv extend-vlan 10
  otv use-adjacency-server unicast-only
  no shutdown


interface Overlay1
  otv join-interface Ethernet1/25
  otv extend-vlan 10

  otv adjacency-server
  otv use-adjacency-server unicast-only
  no shutdown


There is not much special about the config, and you will see that the default packet size you can send is shown below:

N5K7# ping df-bit packet-size 1430
PING ( 1430 data bytes
1438 bytes from icmp_seq=0 ttl=254 time=1.542 ms
1438 bytes from icmp_seq=1 ttl=254 time=1.441 ms
1438 bytes from icmp_seq=2 ttl=254 time=1.187 ms
1438 bytes from icmp_seq=3 ttl=254 time=1.168 ms
1438 bytes from icmp_seq=4 ttl=254 time=1.157 ms

The maximum you can do is 1430, this is because OTV adds 42 bytes of overhead on a typical IP frame, so if 1472 if your maximum, take away 42, that gives you 1430.

Let's try and configure some jumbo MTU's and see how much bigger we can get them to go.

First thing would be to enable Jumbo frames on the SVI's, Duh!

N5K7(config)# int vlan 10
N5K7(config-if)# mtu 9216

Still no dice, next we enable system qos class on the 5k's:

N5K7(config)# policy-map type network-qos JUMBO
N5K7(config-pmap-nq)# class type network-qos class-default
N5K7(config-pmap-nq-c)# mtu 9216
N5K7(config-pmap-nq-c)# exit
N5K7(config-pmap-nq)# system qos
N5K7(config-sys-qos)# service-policy type network-qos JUMBO

Still no dice,

However, if we enable our 7k join interface and M interface for MTU:

int eth2/27
(Interface towards 5k)
mtu 9216
int eth1/25
mtu 9216

Everything works as expected and we can send jumbo MTU.

N5K7# ping df-bit packet-size 2173
PING ( 2173 data bytes
2181 bytes from icmp_seq=0 ttl=254 time=1.825 ms
2181 bytes from icmp_seq=1 ttl=254 time=1.389 ms
2181 bytes from icmp_seq=2 ttl=254 time=1.213 ms
2181 bytes from icmp_seq=3 ttl=254 time=1.344 ms
2181 bytes from icmp_seq=4 ttl=254 time=1.358 ms


As per the brilliant blog post by Tony Burke over at the Data Centre overlords:

You can modify MTU on a per-VSAN basis, and the MTU will be negotiated between the devices during FLOGI.




So, I can tell you now definitively as I have tested it again and again and I know for sure:

So, The most important thing, is the System QOS

This does not suprise me again: The Fabric Interconnect is based off the Nexus 5000, so it makes sense it would work in a similar manner.

The System QOS Class is  the most important value to set: if you do not set this to support your jumbo frames, no matter what you set on the OS, no matter what you set on the vNIC (and in fact you won't be able to set ANYTHING on the VNIC that is higher than the system class) it won't work unless you have set the system class also.

That is found here:

You MUST change these values to 9216 or whatever jumbo MTU you want, this is key, without this you will get NOWHERE!

OK now we have that cleared up, and we have changed the value:

We need to know what this setting, under the vNIC, does here:


 (see highlighted section)

As per Jeff Said So's blog post on the topic: http://jeffsaidso.com/2012/04/cisco-ucs-mtu-sizing-with-vic/

This value is STRICTLY used to INFORM the OS that the network card supports this particular MTU value, if your OS supports auto detection of the network card MTU, you therefore do not have to manually set the MTU in the operating system, which saves time obviously.

Now, _IF_ you set the MTU value manually in the OS, As i will show you below:

 but you leave this value at 1500, as I have done in the below screenshot:

 Guess what? You can get jumbo frames:

So!!! This value in the vNIC is used for one reason and one reason only: to inform the OS that THIS is the available MTU, and _IF_ the OS supports it, it will set the adapter to the appropriate MTU, if you either forget to set this value, OR your OS does not support it, if you change the value in the OS MANUALLY, you will still have jumbo frames as long as the system class is configured.

My Final advice? In the real world Set the jumbo MTU on the adapter, it can't hurt if your OS supports the auto detection, and if your OS does not, well you just need to set the MTU manually.

This explains why the MTU you set on a vNIC does not come up under the actual vethernet config under UCS: It's just a recommendation to the operating system. I would be interested to see what changing this value does on NON Cisco VIC cards, and in actual fact can you even change this value? Any thoughts on this please make a comment.

So, this value is PURELY a suggestion to the OS, it is NOT enforced, but you might as well set it anyway.

The reason this only has a maximum value of 9000, even though you configured 9216 in the system-class, is because most operating system's only understand a value of 9000.

MTU on Nexus 1000v

Luckily this has become extremely simple since 4.0, there IS a system jumbo MTU command, but it does nothing, it cannot be changed and cannot be removed:

 If you try and change it:

Only valid options are to set it to 9000.

alright, so how do I set my jumbo MTU?

Well, you ONLY have to set it on your uplink interfaces, it works BY DEFAULT on all your vethernet interfaces

See below:

So you don't have to set anything, if you try and configure an MTU on a vethernet, the option will simply not be there:

You can only set it on the uplink interface, and this is the only place you need to set it: