Gigabyte BRIX / Intel NUC Gen6/7 – ESXi shutdown issue fixed

I bought some month ago three Gigabyte BRIX GS-BSi3HAL-6100 boxes to run vSAN on top of it in my home lab. These boxes were running 24/7 for the last couple of month without doing anything because I didn’t find any time to play with it and put it into “production”. I was also playing with Host Profiles and Auto Deploy which took also some time. In the end I decided to shutdown these boxes to save some money on the power bill. I’m running these boxes without any keyboard or monitor. During the shutdown I realized that the ESXi host was shutdown but the hardware box was still running. Unfortunately when connecting a HDMI monitor to the running box I saw only a black screen. I decided to connect a keyboard and a monitor to one of the boxes to see what was going on during shutdown.

Issue

When I shutdown the ESXi host I got the following message on the screen with vSphere 6.5d and Update 1:

This system has been halted. It is safe to use the reset or power button to reboot.

I also tried to see if this is only a vSphere 6.5 issue or if it’s also present in 6.0 Update 3. With 6.0 Update 3 the “message” was little bit different.

During shutdown the normal DCUI screen was shown and no key was working. The only different was a small different colored “L” in the down left corner of the white frame.

If you were fast at the shutdown and press ALT-F12 you were able to see the last entries in the vmkernel.log.

VMKAcpi: Bus ffff can’t wake system from S5 (max is S4).

This was indeed an unexpected behaviour because I thought it was working when I first installed the Gigabyte BRIX.

Troubleshooting

BIOS downgrade

First I thought that the most recent BIOS update (F5) had some settings which were changed from the previous one (F4). So I used the AMIBCP Bios Editor to check which settings were new and which were old. Unfortunately without luck. Only one Bios setting, which was removed and replaced with a default value in the newest Bios, but nothing else. To be sure I downgraded one of my Gigabyte BRIX to the F4 BIOS but the issue was still there.

BIOS settings check

I also went through all the BIOS settings which can lead to the message “VMKAcpi: Bus ffff can’t wake system from S5 (max is S4).” But I was here also out of luck.

Reach out to Twitter

I reached out to Florian Grehl (http://www.virten.net/) who is the Intel NUC Homelab Guru and he also told me that he has the same issues with the latest Intel NUC Generation 7. Unfortunately he had also no fix for this issue. During this conversation also Mads Fog, Christian Parker and Mehdi Khodaeifard told me that they have the same issue with their Gen6 and Gen7 NUCs. Florian has also a very good post about the Kaby Lake NUC here but also there in the comments, people having the same problem but with no solution.

Downgrade from vSphere 6.5 to 6.0 Update 3

One of my troubleshooting steps also includes downgrading 6.5 to 6.0 Update 3 to see if this issue also happens there. To my surprise in 6.0 Update 3 this issue doesn’t exist. During this step I didn’t know that I have already changed something which has fixed my problem. This was the reason 6.0 Update 3 was working.

Reaching out to William Lam

When you are out of ideas the last hope you will get is from the awesome William Lam. I reached out to William internally asking him for help. He pointed me to another colleague and told me I should open a Case and upload all the required logs from 6.5 and 6.0 so GSS can check which settings are different between the working 6.0 installation and the non-working 6.5.

Solution

When William told me to open a case I created a support bundle from the working 6.0 Update 3 host. After I got this bundle I installed 6.5 Update 1. To my surprise this was working now too. I thought “Great. I’m back where I started”. Now I took the USB drive where the original 6.5d was installed and also this was working now.
I checked the other boxes if they were also working now but “thanks god” both of them still had the issue. So what did I changed in the last hours that could fix the problem. As far as I can remember I changed the following:

• Created a VMFS volume and redirected the log files there
• Bios setting “Onboard Audio” to Enable

To evaluate which of these settings was the fix I redirected the log files from the VMFS volume back to the /scratch/log. Afterwards I deleted the local VMFS datastore. After the ESXi restart and shutdown the problem was still not there. So the next step was to disable the “Onboard Audio” again and to my suprise the problem is back again.
I also checked this behaviour with 6.0 Update 3 and 6.5 Update 1 and in both cases disabling the onboard audio led to this issue.
Unfortunately I can’t tell why disabling the onboard audio leads to this issue. In addition I can only confirm that this is the fix for my Gigabyte BRIX boxes. I don’t know if this is also the fix for the Intel NUC Gen6/7. So everybody who has currently the same problem, please check if you also have “Onboard Audio” disabled.
Here is a screenshot where you can find the settings on the Gigabyte BRIX BSi3HAL-6100.

If you have some feedback to this post or info about the Intel NUC Gen6/7 fix please leave a comment.

Update 2017-11-03

It looks like that this fix is specific for the Gigabyte BRIX. It looks like that the Intel NUC Gen6/7 have another problem.

Update 2018-04-27

It looks like this problem was solved in vSphere 6.7 for the NUCs.