Joanna Rutkowska, a Polish researcher at Singapore-based Coseinc, showed that it is possible to bypass security measures in Vista that should prevent unsigned code from running.
And in a second part of her talk, Rutkowska explained how it is possible to use virtualization technology to make malicious code undetectable, in the same way a rootkit does. She code-named this malicious software Blue Pill.
"Microsoft is investigating solutions for the final release of Windows Vista to help protect against the attacks demonstrated," a representative for the software maker said. "In addition, we are working with our hardware partners to investigate ways to help prevent the virtualization attack used by the Blue Pill."
At Black Hat, Microsoft gave out copies of an early Vista release for attendees to test. The software maker is still soliciting feedback on the successor to Windows XP, which is slated to be broadly available in January.
Rutkowska's presentation filled a large ballroom at Caesars Palace to capacity, even though it was during the last time slot on the final day of the annual Black Hat security confab here. She used an early test version of Vista for her research work.
As one of the security measures in Vista, Microsoft is adding a mechanism to block unsigned driver software to run on the 64-bit version of the operating system. However, Rutkowska found a way to bypass the shield and get her code to run. Malicious drivers could pose a serious threat because they run at a low level in the operating system, security experts have said.
"The fact that this mechanism was bypassed does not mean that Vista is completely insecure. It's just not as secure as advertised," Rutkowska said. "It's very difficult to implement a 100 percent-efficient kernel protection."
To stage the attack, however, Vista needs to be running in administrator mode, Rutkowska acknowledged. That means her attack would be foiled by Microsoft's User Account Control, a Vista feature that runs a PC with fewer user privileges. UAC is a key Microsoft effort to prevent malicious code from being able to do as much damage as on a PC running in administrator mode, a typical setting on Windows XP.
"I just hit accept," Rutkowska replied to a question from the audience about how she bypassed UAC. Because of the many security pop-ups in Windows, many users will do the same without realizing what they are allowing, she said.
Microsoft has touted Vista as its most secure version of Windows yet. It is the first operating system client to go through the company's Security Development Lifecycle, a process to vet code and stamp out flaws before a product ships.
"Windows Vista has many layers of defense, including the firewall, running as a standard user, Internet Explorer Protected Mode, /NX support, and ASLR, which help prevent arbitrary code from running with administrative privileges," the Microsoft representative noted.
After the presentation on bypassing the driver shield, Rutkowska presented a way to create the stealthy malicious software she code-named Blue Pill. The technique uses Pacifica, a Secure Virtual Machine, from chipmaker Advanced Micro Devices, to go undetected.
Blue Pill could serve as a backdoor for attackers, Rutkowska said. While it was developed on Vista and AMD's technology, it should also work on other operating systems and hardware platforms. "Some people suggested that my work is sponsored by Intel, as I focused on AMD virtualization technology only," she said, adding that is untrue.