Hey Dude where’s my NIC?

Today I was at a customer site and the customer told my a really strange story and I don’t know if everybody else is/was aware of the following behavior. But first of all the story.
He was contacted because a Terminalserver was no longer reachable by the IP address. He looked at  the VM settings and found out that the NIC was gone. He added a NIC back to the VM and the Terminalserver was reachable again. This happened 2 weeks ago and he told me the problem today. So this story immediately awakened my curiosity and troubleshooting ambition.
First I checked Task/Events to see if there was a manually removal within the vSphere Client. but I found nothing. The only thing I found was the entry where the customer added back the missing NIC. After that I downloaded the vmware-##.log from that VM at a given period and found the following:

Powering off Ethernet0
Hot removal done.

My first thought was “OK….. and who did that?”. I did some research and found this VMware Blog post. The following information is taken from this post.
There are 2 reasons why a NIC go missing from the VM.

• Hot Removal from the Guest. With the new Hot Add/Remove feature, NICs show up under the “Safely Remove Hardware” list. Any user with administrative privileges can accidentally remove the NIC using this feature.

• Someone manually removed it from the Virtual Machine configuration (Probably using UI or some SDK APIs).

If somebody removes the NIC through vSphere Client or SDK APIs you will find the following in the vmware-##.log within the Virtual Machine folder:

Requesting hot-remove of ethernet1

If the NIC is removed within the guest OS using “Safely Remove Hardware” you will see the messages mentioned earlier. What I didn’t know was the fact, that if the NIC is  removed in that way it is also removed from the VM settings! I thought until today that this function will only remove the NIC from the guest OS and will be back after a reboot.
There are 2 ways to avoid this problem.

1. Disable the HotPlug capability of the VM

If you disable HotPlug of the entire VM you can’t add HDD and NICs if the VM is powered-on.

1. Connect to the ESXi/ESX host or vCenter Server using the vSphere Client.
2. Power off the virtual machine.
3. Right-click the virtual machine and click Edit Settings.
4. Click the Options tab.
5. Click General > Configuration Parameters > Add Row.
6. Insert a new row with the name devices.hotplug and a value of false.
7. Power on the virtual machine.

When you try to add an hard disk or network card you will get this:

2. Use Windows Registry hack to hide the hot removeable capabilities of the NIC

The post I mentioned previous is from 2010 therefor some procedure are outdated for Windows 2008 R2/2012/2012 R2. To edit the desired value you have to start the registry editor as Local System. To do so download either the complete Sysinternals Suite or only the psexec Tool from the Microsoft Sysinternals site.

1. Open a Command Line Prompt (cmd)
2. Execute psexec -i -s regedit.exe
3. Go to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum and search for your NIC driver (VMXNET3, E1000 etc.)
4. Set the Capabilities flag in the key(s) found above, to the current value minus 4

For example I have the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_15AD&DEV_ 07B0&SUBSYS_07B015AD&REV_01\FF5650006DFF91FE00 (1) with the Service value (2) vmxnet3ndis6. The capabilities value is set to 16 (3). On changing the value to 12, (immediately) VMXNET3 NIC will be no longer listed in the safely remove hardware list anymore.

After Capabilities changed to 12

NIC is hidden from “Safely Remove Hardware”