vSphere 5.5 Upgrade and old NICs not working as expected
I upgraded one of my home lab ESX to 5.5. After that I recognized that network connectivity on my Distributed switch was lost. After a quick research I found out that my vmnic which is connected to the vDS didn’t receive any CDP packets. I found a KB Article that support for older NICs was dropped when upgrading to 5.5. This doesn’t mean that this NICs won’t work, because the driver is still installed but are not longer supported by VMware.
To get your exact physical NIC name you can use either the vSphere C# Client or the vSphere Web Client.
vSphere C# Client
vSphere Web Client
If you need the Device ID or Driver information of your NIC you can login to your ESX host through SSH and leverage the following commands:
vmkchdev -l (Device ID)
ethtool -i vmnic1
Searching the list which is includes in the stated KB article I found my NIC around the unsupported ones.
I tried some additional configuration with this NIC (switch to vSS etc.) and after another troubleshooting session I found the root of the problem. My vDS is configured to use Jumbo Frames (MTU 9000), because I was running VXLAN for my VCAP-CIA exam preparation and the network card is also capable of jumbo frames. When I use the normal MTU size (1500) everything works fine. I tested it till MTU size 8175. Everthing beyond that doesn’t work in that configuration. I got a short twitter discussion with @Andrea_Mauro and @vPatrickT. Both told me to upgrade the firmware of the NIC. Now that was another problem. The NIC I used was a HP NC7170 dual port card which was produced in 2004/2005 (no guarantee that the year is correct), so it’s an old one and for that card there is no firmware upgrade.
But can an old firmware with new driver causes jumbo frames not to work or in other words not correctly to work? As mentioned packets until MTU size 8175 worked.
My answer to this question is simply “I don’t know”. There are only a few moments where I have no answer to a problem, not even a hint and this is such moment. So anybody who will read this post leave a comment if you have any idea regarding this problem. I really will appreciate it.
The workaround for me was to use a complete different quad port NIC which is supported with 5.5 and lo and behold Jumbo Frames worked again.
UPDATE
A comment from Martin Knaup (www.vaspects.com) gave me some information about an updated firmware of my Dual Port NIC. I downloaded the lastest version v19 and put the Dual Port NIC in one of my “testing” PCs to flash the new firmware. Martin Knaup described the procedure for a Quad Port card in his post.
After the download extract the File wherever you want. The default location for the self-extractor is C:\Intel19.0\. Navigate to C:\Intel19.0\APPS\BootUtil\Win32. There you will find the BOOTUTILW32.exe which is the main flash executable.
Run the following command to get the current version of the NIC firmware
BOOTUTILW32 -IV
In my case the NIC has a firmware version of 1.2.11 which is quite old. To upgrade the firmware I have to place the NIC into flash mode which can be done with the following command.
BOOTUTILW32 -all -fe (all is for all NIC ports and fe is for flash enable)
After a reboot you can run the command with the update parameter.
BOOTUTILW32 -all -up=combo -file=..\BootIMG.FLB (up is for update and combo describes that the Image which is used is a Combo file incl. all the supported chipsets)
As you can see both ports were updated to 1.5.50 which is the most recent firmware for this card.
There are several other command parameters which can be used on this card. Here is a short overview.
The only thing which didn’t work on my chipset was the Wake-on-LAN parameter. I don’t have any clue why this is an unsupported feature, because in the description of the chipset it states WOL capability.
BOOTUTILW32 -all -wole
After I upgraded the card I put it in my old ML115 G5 and installed vSphere 5.5. I added the server to my vDS but unfortunately it still didn’t work. Just for sure I updated the server to 5.5 Update 1, but still no success with Jumbo Frames. It really looks like that something fundamental has changed that these cards are unsupported with vSphere 5.5.
Hi Manfred,
you may try the “Intel Ethernet Connections Boot Utility, Preboot images, and EFI Drivers”:
https://downloadcenter.intel.com/Detail_Desc.aspx?DwnldID=19186
From the PCI IDs (Vendor ID Intel 8086) I would expect them to work fine, at least this was the case with my HP and IBM OEM versions of the i340-T4 quad port NIC and also before with the dual port HP NC360T NICs (which I replaced with quad ones). But to be perfectly honest I have no idea if this will replace other portions of the firmware besides the option ROM. I used an USB stick with a Windows PE environment to update the firmware on my ESXi hosts:
http://www.vaspects.com/2013/11/22/choose-the-right-quad-port-nic/
Regarding the Jumbo frame problem I had the same issue with the onboard Realtek NICs, but in contrast to your Intel chips these are known to be buggy to the core and won’t work properly with anything more than 7000 bytes or so.
BR Martin
Hi Martin,
thanks for your comment. I will try this and get back to you with my results!
Thanks.
Cheers
Fred
Hi Fred!
Great, please let me know. I just updated my NICs with v19.0, no problem. Besides a bug in the Debian 7.4 igb driver, but the ESXi run fine.
BTW I just had some “fun” with Jumbo frames in another environment as well. Every time I started a new VM on an iSCSI datastore (on a Thecus NAS) the network traffic went ballistic, basically shutting down all connectivity. All other operations, incl. Storage vMotion, worked flawlessly. “Something” in the setup caused iSCSI timeouts followed by fast path state updates by the ESXi. Turned of Jumbo frames – all good again. Strange.
Hi Martin,
upgraded my old NIC to v19.0 but it didn’t change anything. Same behavior like before!
Excellent article. I’m dealing with many of these issues as
well..