[attrition] rant: How a PR Miracle can save VMware from itself

[lyger]

http://attrition.org/security/rant/vmware/

How a PR Miracle can save VMware from itself
(And how one little itsy-bitsy code snafu has resulted in a huge FAIL for 
the soon-to-be AMD of the virtualization industry.)

Sat August 16 19:37:01 EDT 2008

By: martums

On or about 12 August 2008 at 1238 hours EST AU , VMware Communities user 
mattjk of Melbourne, Australia, started a thread in the VI3 ESX 3.5 
Communities Discussion Forum. This thread was one of the earliest public, 
high-profile indicators of the failure of ESX & ESXi 3.5 update 2 to 
power-on or VMotion virtual machines, aka guests, if the ESX (or host) 
server time was on or after 12 August. Effected builds include 103908 & 
103909, respectively ESX and ESXi. Without VMotion, features leveraged by 
Enterprise customers such as High Availability (HA) and Distributed 
Resource Scheduler (DRS) also failed, potentially crippling some VMware 
shops.

Within 24 hours, over 500 responses to the original post were added. One 
by one, as clocks in each successive time zone moved into Tuesday morning, 
the early adopters of 3.5 update 2 around the globe began to encounter "A 
General System error occurred: Internal error" in their Virtual 
Infrastructure (VI) client applications. As the numbers of host failures 
increased globally, the impacts of this incident had several obvious 
effects on the vendor:

     * The communities thread started by mattjk grew quickly, with hundreds 
of replies and tens of thousands of views.
     * Concurrently, those same early adopters flooded VMware Support with 
telephone calls.
     * The web servers hosting the VMware knowledge base were effectively 
slashdotted. Behold...

The First Global Hypervisor Failure

Initial indications suggested that VMware erroneously released their 
latest ESX & ESXi server product builds with some sort of time-limitation 
which impacted the ESX host's license management. This release of ESXi was 
the first time any edition of ESX, (in this case, just ESXi), had been 
made freely available, in an earlier announcement by Paul Maritz, the new 
boss, head man, top dog, big cheese. Free, as in (light) beer. (I realize 
that the rule is never complain about free beer. But this one tastes 
terrible).

[...]