ESX 3.5 Update 2 bug

Many customers are hitting a new bug in ESX preventing them from powering on virtual machines or VMotion them.  There is a published knowledge base article on this which reads>>

Unable to Power On virtual machine with “A General System error occurred: Internal error”

A virtual machine fails to power on, fails to leave suspend mode, or fails to migrate with VMotion and the following message is displayed in the vmware.log file for the virtual machine:

This product has expired. Be sure that your host machine’s date and time are set correctly.
There is a more recent version available at the VMware web site:
https://www.vmware.com/info?id=4.
————–
Module License Power on failed.

Currently running virtual machines will continue to run but will not be able to perform migration with VMotion, suspend or power on virtual machines.

An issue with ESX/ESXi 3.5 Update 2 causes the product license to expire on August 12, 2008. VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files by noon, PST on August 13.  These will be available from the page: https://www.vmware.com/download/vi.  Until then, VMware advises against upgrading to ESX/ESXi 3.5 Update 2.

The Update patch bundles will be released separately later in the week.

This KB article will be updated as soon as more information is available, check back frequently for updates and additions.

A workaround to this ESX bug is to set the date back a day on your ESX hosts and you will be able to power on the VM’s again. If you use ntp, disable that for the time being and manually set your hosts back a day.  Be careful, if you have your VMware tools configured to sync VM time to the host time then this will alter your VM times.

***UPDATE***

Here is the latest public news from VMware on this>>

Problem:

An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully.  This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2.  The problem is caused by a build timeout that was mistakenly left enabled for the release build.

Affected Products:
-    VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2 (pre-Update 2 releases are not impacted by this problem).
-    Reports of problems with ESX 3.5 U1 with the following patches applied.
1.    ESX350-200806201-UG
2.    ESX350-200806202-UG
3.    ESX350-200806217-UG

-    No other VMware products are affected.

What has been done?:

-    Product and Web teams pulled the ESX 3.5 Update 2 bits from the download pages last night so no more customers will be able to download the broken build.
-    VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
-    A Knowledgebase article has been published (https://kb.vmware.com/kb/1006716), but traffic to the knowledgebase is causing time outs.  A new static page has been  published at https://www.vmware.com/support/esx35u2_supportalert.html that customers and partners will be able to view.
-    The phone system has been updated to advise customers of the problem
-    Vmware partners have been notified of the issue.

Workarounds:
1)    Do not install ESX 3.5 U2 if it has been downloaded from VMware’s website or elsewhere prior to August 12, 2008.
2)    Set the host time to a date prior to August 12, 2008.  This workaround has a number of very serious side affects that could impact product environments.  Any Virtual Machines that sync time with the ESX host and serve time sensitive applications would be broken.  These include, but are not limited to database servers, mail servers, & domain administration systems.

Next Steps:
VMware to send an email to all customers who have downloaded this version. This effort is underway and should happen before 11 am today.

Resolution:

VMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today.  The target timeframe is 6pm, August 12, 2008 PST.

FAQ:

1. What would this express patch do?

More information will be provided in subsequent communication updates.

2.   Will VMware still reissue the upgrade media and patch bundles in the timeframe that has been communicated?

Yes.  We still plan to reissue upgrade media by 6pm, August 13 PST (instead of noon, August 13 PST) and all update patch bundles later in the week.  We will provide an ETA for the update patch bundles subsequently.  NOTE: the “patch bundles” referred to here are for the patches listed above under “Affected Products” and the other bundles released at GA.  They are not the same as the express patch which is targeted for 6pm, August 12, 2008 PST as stated above.

3.   Why does VMware plan to reissue the upgrade media before the patch bundles?  That is a wrong priority call!

This is not a matter of priority.  Since we can get done building and testing the upgrade media before the patch bundles, we want to make that available to customers first instead of reissuing all the binaries later in the week.

4.   Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?

There is no licensing backdoor in our code.

5.   Does this issue affect VC 2.5 Update 2?

No.

6.   What is VMware doing to make sure that the problem won’t happen again?

We are making improvements on all fronts.  The product team had endeavored to deliver a release with support customers deem important.  But we fell short and we are deeply sorry about all the disruption and inconveniences we have caused.  We have identified where the holes are and they will be addressed to restore customers’ confidence.