[ISN] Get Ready To Patch

[InfoSec News]

http://www.informationweek.com/story/showArticle.jhtml;jsessionid=QW0L2OWLDGIHWQSNDBCSKHQ?articleID=45400083

By John Foley and George V. Hulme 
Aug. 30, 2004 

In the first three weeks it's been available, businesses have
downloaded more than 1 million copies of Microsoft's Service Pack 2
for Windows XP, and consumers have downloaded many more. It's merely a
start in what's shaping up to be the most far-reaching and complex
software patch ever attempted. Over the next three months, Microsoft's
goal is to push SP2 out to more than 100 million PCs. Few expect it to
be easy. Applications already are breaking as software vendors and
systems administrators test the security-packed Windows update before
rolling it out to users. Microsoft has identified about 50
applications that are incompatible with SP2, and company officials
admit that many custom applications are likely to encounter glitches,
too. Last week, Microsoft released a 100-page technical document that
describes how companies should assess applications for compatibility
with SP2 and what they should do when things don't work.

Microsoft's mother of all patches is just the latest in what's become
a familiar and frustrating industrywide exercise, as software
companies and their customers race to stay ahead of the worms and
other attacks that seek to take advantage of newly discovered
vulnerabilities in operating systems and applications. "You have to
take this stuff seriously. You can't let your guard down for a
second," says Michael Kamens, global network and security manager with
Thermo Electron Corp., which has tested SP2 but hasn't determined a
rollout schedule for its several thousand Windows XP machines.

For many companies, patching has been akin to software triage, with IT
personnel dropping what they're doing every time a critical security
bulletin rings the alarm. A growing number of companies, however, are
putting people, processes, and tools in place to bring greater
efficiency and control to that ad hoc way of doing things. And
technology vendors are making some much-needed changes, too.

Oracle has revealed that it will begin releasing its software patches
on a once-a-month schedule, so customers can better plan for them. "We
believe a single patch encompassing multiple fixes on a predictable
schedule better meets the needs of our customers," Oracle said in a
written statement. Oracle also indicated that a security fix would be
issued shortly for vulnerabilities that have been discovered in its
products but declined to comment further on the pending fix or its
revised patch strategy.

Microsoft began issuing monthly patches last October, and Computer
Associates and SAP have been on regular schedules even longer. SAP
uses its Support Portal to make updates available, including
specialized patches for customers who may need help reconciling SAP
applications with third-party products. CA delivers patches once a
quarter, but it moves faster when necessary. "When I sit down with
customers, I seldom get to bring up the issue--it's usually one of the
top things they mention," says Sam Curry, VP of CA's e-Trust
security-management unit.

Jim Burdiss, VP and CIO of Smurfit-Stone Container Corp., likes the
trend toward scheduled patches. "The end game is to get away from fire
drills as much as possible," he says. "When those patches happen
randomly, you force IT to go into a reactive mode." The randomness of
ad hoc patches makes resource and budget planning difficult, he says.

Oracle's policy change and product improvements from Microsoft,
including new features in Systems Management Server 2003 that automate
aspects of Windows patch management, are steps in the right direction.  
But challenges remain. The Yankee Group consulting firm estimates that
a company with more than 500 PCs spends up to 120 staff hours testing
and installing every patch. "The issue is, companies have to test and
test before deploying a patch," says Yankee Group senior analyst Eric
Ogren.

At the Arkansas Army National Guard, two people work full time
patching about 50 Windows servers and 1,500 PCs. "That seems
excessive," says senior network manager Lynn Melton. "It's
frustrating." The military unit uses several tools to deploy patches,
including St. Bernard Software's UpdateExpert, Lieberman Software's
User Manager Pro, and Cisco Systems' CiscoWorks. Melton tried an
earlier version of Microsoft's Systems Management Server but it
required too much effort, he says. He's interested in the vendor's
Windows Update Services patch-management system, which promises to let
customers handle patches for more products than Windows, including SQL
Server and Exchange. But it won't be ready until the first half of
next year. "If we could use one tool to do more than one thing, that
would be helpful," Melton says. Thermo Electron uses Microsoft's
Software Update Services 1.0 tool (the predecessor to Windows Update
Services) for patching at its headquarters, but remote locations
continue to handle the job locally, so it's a challenge to get
everything done quickly. "The problem is, you need a dedicated
full-time person to write scripts and push the patches out there,"  
security manager Kamens says. The company is deploying Systems
Management Server 2003 to help, but at an estimated total cost of
about $1 million, it won't be cheap. Even after predeployment testing,
Kamens says, patches too often "break things." But it's something that
has to be done--the risks of unpatched systems include worms and other
threats, the data vulnerabilities and system snags associated with
such threats, and potential liability, lost productivity, and other
costs related to any security breaches. Thermo Electron's IT staff
rolls out software updates to 800 servers once a month on a Sunday
morning to minimize system downtime.

Companies of all sizes are grappling with the issue. Ajacs Die Sales
Corp., a small distributor of tool-and-die components, has only VP of
IT Steve Wierenga to patch its 22 PCs and four servers. "We have it
under control," says Wierenga, who evaluates Microsoft's patches
himself each month. "We're small enough that we can address an issue
with a patch in short order if it causes a problem." At the other end
of the spectrum, software vendor SupportSoft Inc. says one of its
customers, a bank with 50,000 PCs, will have dozens of technicians
testing the SP2 patch over several months.

Stolt Sea Farm, a seafood company, takes a no-frills approach. The
company's IT environment consists of 550 thin-client terminals and 50
Windows servers spread among locations in about a dozen countries.  
Because there are no desktop PCs to support and most of its software
comes from Microsoft, the company's small IT staff is able to install
patches within 24 hours--and it does so without any testing. "I would
say we are very efficient," says systems administrator Terje Sorgjerd.

CIO Burdiss of Smurfit-Stone Container believes businesses need to
master the nuts and bolts of patch management to focus IT resources on
what really matters: delivering increased business value. "Before you
can do governance and develop the value of IT to the business and all
of the things we're trying to aspire to, you have to have some
credibility," he says. "In my mind, the lights-on stuff has to work
every time, and these patches can be counter to that."

The good news is that companies generally seem better prepared to deal
with patches today than a year ago, using patch-management products
from specialists such as PatchLink Corp. and Shavlik Technologies LLC
and new capabilities from their primary software suppliers. For
example, PeopleSoft Inc., which issues patches quarterly, has cut the
number of manual steps required to find, download, and install patches
and software updates from 49 to seven.

Better defined internal procedures at user companies are helping, too.  
As a result, the Yankee Group estimates costs have dropped to about
$150 per patch for each PC, from about $250 last year. Companies are
"better at it than they were 12 months ago," says Michael Cherry, an
analyst with Directions On Microsoft. "But it still requires a
considerable allocation of resources."

That will be especially true with SP2, which, at a minimum of 75
Mbytes per machine, promises to clog networks if not managed
carefully. And once it's installed on PCs, help-desk administrators
could see a spike in support calls as users grapple with nuances in
the way Microsoft's Internet Explorer browser works with SP2 and other
security-related changes. "It's going to cause as many problems as it
fixes," predicts Simon King, SupportSoft's director of product
marketing for enterprise solutions. "It's going to be a huge
undertaking."

Microsoft group product manager Barry Goffe says the company is doing
everything it can to help. In addition to the 100-page
applications-compatibility document, it has already released a
200-page technical overview of SP2, a Solution Accelerator that
provides guidance on how to load Windows XP SP2 onto a computer, and
other documentation. Over the next few months, Microsoft plans to
deliver the beta version of an applications-compatibility toolkit for
SP2, which will automate some manual processes. And next year,
improved patch management in the form of Windows Update Services
should arrive.

It makes for quite a patch. The next few months will tell just how
much companies have really improved at managing it all.

-- With Charles Babcock and Beth Bacheldor