While my upgrade from vCloud Director 1.5.1 to 5.1 went on through out the day, I started to have a sinking feeling that I wasn’t going to be able to complete it with zero downtime for all of the VMs in the environment.

In our environment, a lot of training and product demos happen, and much of that relies on utilizing nested ESXi, similar to how VMware’s Hands On Labs are run at VMworld (and thankfully, now available online outside of the event).

William Lam has a great article on modifying your vCloud Director database to automatically pass the ‘nested hypervisor’ support flag to vCloud hosts as they’re brought into vCD to be used as a resource rather than having to modify each vSphere hosts’s config file.  http://www.virtuallyghetto.com/2011/10/missing-piece-in-creating-your-own.html

However, with vSphere 5.1, VMware changed how nested ESXi is enabled. It’s now on a per VM basis rather than a per host basis. William’s post “How to Enable Nested ESXi & Other Hypervisors in vSphere 5.1” covers the changes and the new process quite well, so I won’t cover that here.

The biggest kicker to this is that it requires the VM being VMware Hardware Version 9 which is new to vSphere 5.1. So, any current nested ESXi (or any other nested hypervisor) is running, at highest, Hardware Version 8.

In order to power on a VM with Hardware Version 9, a vSphere host must be at vSphere version 5.1. However, a host with vSphere 5.1 does not support a VM using CPU features that were only flagged as available from a vSphere configuration. The result was the following error when a vMotion was attempted from a vSphere 5.0 host to a 5.1 host with one of these nested VMs:

The VM failed to resume on the destination during early power on.
This machine or version of VMware ESX does not support the virtualized Intel VT-x/EPT features required by this virtual machine. Resume the checkpoint on a machine that supports the required virtualized Intel VT-x/EPT features.

Unfortunately, the only way to upgrade a VM’s Hardware Version is to shut it down. Once you do, you can migrate the VM to a vSphere 5.1 host, upgrade the Virtual Hardware, then expose hardware-assisted CPU virtualization to guestOS. This is a bit different on an existing VM than from creating a new one as the CPU flag is under the ‘Hardware’ tab in vCloud Director.

The first step, after you’re finished with your vCloud Director and vCenter Server upgrades, is to change the “Highest supported hardware version” at the Provider vDC level:

Then you can power down your nested hypervisor, migrate it to a vSphere 5.1 host (you’ll have to do this via vCenter), and then upgrade the HW version (showing here via vCloud Director).

You’ll need to hit ‘OK’ to upgrade the Hardware Version before you can then go back and check the CPU flag box.

Once these settings are in place, you can go on nesting and incepting the virtual world.