[Infowarrior] - Alert: Some VMWare machines going "boom!" tomorrow

[Richard Forno]

(Reposted from the blog in its entirety since it may be Slashdotted)  - 
rf


http://www.deploylinux.net/matt/2008/08/all-your-vms-belong-to-us.html

All your VM's belong to us
By
Matthew Marlowe
on August 11, 2008 9:48 PM | Permalink | Comments (4) | TrackBacks (0)


As of tomorrow morning, VM's running on all hosts with ESX 3.5U2 in  
enterprise configurations will not power on.

Boom.

Apparently, there is some bug in the vmware license management code.  
VMware is scrambling to figure out what happened and put out a patch.

There is a major discussion going on in the vmware communities about  
it: http://communities.vmware.com/thread/162377?tstart=0

OK, while we're all remaining calm....just imagine the implications  
that bugs like this can occur and get past QA testing....5 years down  
the road, nearly all server apps worldwide pretty much running in VM's  
(pretty easy prediction)......some country decides to initiate  
cyberwarfare and manages to get a backdoor into whatever is the  
prevaling hypervisor of the day.....boom. All your VM's belong to us.

I honestly think a lot of the new hype about products dedicated to the  
new industry of vm security is crap, but honestly -- god protect us if  
the baseline code for critical hypervisors like ESX isn't kept secure  
and regularly audited.

I'd love to find out what happened here. Don't they do any regression  
testing on new releases to check for date based bugs? I thought that  
would be pretty obvious.

UPDATE: Frank Wegner has posted the following suggestions:

     You can see the latest status here: http://kb.vmware.com/kb/ 
1006716 Please check back often, because it will notify you when this  
issue has been fixed. Until then the best workaround I can think of is:

     * Do nothing
     * Turn DRS off
     * Avoid VMotion
     * Avoid to power off VM's

I'd council against turning DRS off as that actually deletes resource  
pool settings....instead, set sensitivity to 5 which should  
effectively disable it w/ minimal impact.

UPDATE 2: VMware Website appears to be having trouble keeping up with  
people requesting updates.

UPDATE 3: VMware has stated they will have fixes available in 36hrs at  
the earliest.

UPDATE 4: Anand Mewalal comments:

     We used the following workaround to power on the VM's.
     Find the host where a VM is located
     run ' vmware-cmd -l ' to list the vms.
     issue the commands:
     service ntpd stop
     date -s 08/01/2008
     vmware-cmd /vmfs/volumes/
     service ntpd start

UPDATE 5: Apparently, there are no easily seen warnings in logs/etc or  
VC prior to hitting the bug. VC will continue to show the hosts as  
licensed and no errors will appear in vmkernel log file until you try  
to start up a new vm, reboot a vm, or reboot the host.

UPDATE 6: Welcome Slashdot readers! I've temporarily disabled comments  
to allow the server vm to handle the load. Apparently Movable Type 4.1  
executes a seperate perl cgi script to handle comments on each page  
load. Load times might have been slow for the last 45 minutes, but  
should be OK now.