[Infowarrior] - The rise of zero-day patches

[Richard Forno]

Original URL: 
http://www.theregister.co.uk/2007/03/02/zero-day_patches_interviews/
The rise of zero-day patches
By Federico Biancuzzi
Published Friday 2nd March 2007 13:01 GMT

Interview Zero-day exploits were once the realm of just underground and
elite hackers, but their increased prevalence is bringing a positive new
trend: unofficial patches from members of the community, offered for
protection before official vendor patches appear.

Federico Biancuzzi interviewed Landon Fuller, who wrote Mac OS X patches for
recent Month of Apple Bugs vulnerabilities, and the ZERT team, which has
offered patches for critical Microsoft Windows zero-days that were actively
exploited.
Part 1: Landon Fuller on patching Mac OS X

Could you introduce yourself?

Landon Fuller: I'm a 24 year old bicycle addict. I'm also the director of
infrastructure for Three Rings Design (http://www.threerings.net), a small
video game company based here in San Francisco.

Why have you chosen to write zero-day patches?

Landon Fuller: Writing these patches (http://landonf.bikemonkey.org/)
provides challenging, discrete problems that are fun to solve. Additionally,
I personally appreciate having the option to apply a patch while waiting for
a vendor fix - that's an option that I'm happy to provide to others.

How much time do you need to develop a patch?

Landon Fuller: Generally speaking, it takes anywhere from two to eight
hours. My steps for creating a patch are:

   1. Recreate the test case and reproduce the issue. For example, to
reproduce the Java GIF vulnerability
(http://www.zerodayinitiative.com/advisories/ZDI-07-005.html) from the
provided description, I had to read up on the GIF format and adapt
ImageMagick to produce specifically broken image files.
   2. Find the vulnerable code. This is generally a straight-forward
debugging exercise, but not always - in patching the Flip4Mac
(http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-0466) vulnerability,
I found that Telestream had stripped all but a few symbols from their
binary. Finding the vulnerable code required working backwards up the
soon-to-be mangled stack on my x86 system, locating and labeling functions
according to their assumed purpose and finding the bug (unexpected signed
integer overflow where an unsigned integer should have been used). I then
had to do the same thing on my PowerPC system - considerably simpler once
I'd already found the issue. William Carrel (http://blog.carrel.org/) also
helped debug that one.
   3. Determine the safest way to patch an issue and implement the patch.
Sometimes this can be tricky. With the CoreGraphics PDF infinite-loop DoS,
two functions had to be patched - an entry function, where a per-thread
counter was initialised, and an internal function, where the counter was
incremented. When the counter function hit the maximum loop count, the
patched code returned an error its caller could handle, and CoreGraphics
recognised the PDF as corrupt. Additionally, I implemented an idea suggested
by William Carrel - to protect against recursion, the outer counter
initialiser maintained a reference count that was incremented each time it
was called, and decremented directly prior to exiting. The loop counter was
not deallocated until the reference count hit 0.
   4. Test the patch. Prior to releasing a patch, I would test the fix on
both x86 and PowerPC systems. I also implemented a regression test suite,
and used that to test the patches for all previous issues as well.
Occasionally testing would find an issue, but for the most part, this was
the quickest step.

How much additional effort is needed to support localised versions of the
software?

Landon Fuller:

Fortunately, Apple's means of localisation does not require implementing
multiple versions of a single piece of software. Localised resources are
independent of the code, and in nearly all cases the same patch can be used
for any localisation.

What type of problems (if any) did you have to handle while dealing with two
types of architectures (PowerPC and x86)?

Landon Fuller: Apple (via NeXT) has solved the multiple architecture issue
very nicely - the i386 and PowerPC releases of Mac OS X are built, for the
most part, from the same (or very similar) code base.

The patch was written in C and cross-compiled for both architectures; after
reverse engineering and implementing a patch on x86, I would verify the
function signatures on PowerPC and run the regression tests. Only a few
instances - such as the Flip4Mac patch - required any architecture-specific
wrangling.

Did you use the available source code of Darwin to solve any of the bugs?

Landon Fuller: While none of the non-kernel Apple bugs were in available
source, I did do some research on issues using the available Mac OS X source
- one example being the CoreFoundation %n format string specifier.

Does the MacOS X license limit your power to develop and/or distribute
fixes?

Landon Fuller: I'm not a lawyer, and that's a complex area of law. That
said, I believe that this kind of work is protected under existing US law,
and even more importantly, is ethically sound practice.

How do you install your patches? Do you need to include some pieces of the
file you are going to patch? Do you think this could become a legal problem?

Landon Fuller: The patches are installed as plugins to Application Enhancer
(http://www.unsanity.com/haxies/ape/).

The code is loaded into vulnerable applications at runtime; it finds and
patches the vulnerable functions by name, whether they are in libraries,
plugins, or the applications themselves. This means that the same patch can
generally be used on both PowerPC and x86 systems, no files are ever
modified on disk, and also, the contents of the original file are not
required.

Sometimes the symbols are stripped and can't be found by name - this was the
case with Flip4Mac. I implemented the patch by finding one of the few public
symbols that are exported by the plugin, and then computing the offset to
the vulnerable code on both PowerPC and x86 systems. At runtime, the patch
does the same thing I did manually - it locates the address of the public
symbol, adds the offset I computed, and patches the function at the
resultant address.

Each patch is locked to the specific, latest version of the software in
question, and refuses to patch newer versions - as new software releases are
installed, there's no worry about patch incompatibilities causing crashes,
or more importantly, getting in the way of an official vendor fix.

Have you involved Apple during the process of developing or testing a patch,
or shared your findings after the patch was released?

Landon Fuller: I've made all of the code to the patches available to the
public, including Apple, and I've also privately reported to the vendor any
additional issues I've found in the course of producing those patches.

Have you looked into Apple's official patches for these bugs? From your
analysis, what can you say?

Landon Fuller: The fixes will generally not be entirely the same, as
run-time function patching necessitates wrapping the vulnerable code and
preventing failure conditions from occurring. An official patch will simply
fix the root issue.

Generally vendors take a long time to release patches. They often claim that
this time is required to test all possible configurations and interactions
with other software to avoid breaking production systems. How can your
approach be faster doing reverse engineering?

Landon Fuller: Vendors typically have much more extensive testing, review,
and verification processes in place; they often support multiple released
versions of the software, and multiple products that need to be reviewed for
the same flaw. Releases need to be tested for regressions and public release
needs to be scheduled. These processes are in place to ensure that customers
are well-served by stable software releases.

It may be that development/release process changes could reduce turn-around
for security issues that's difficult to evaluate from the outside. I do
think it's clear, however, that releasing patches for critical issues should
take days or weeks, not months.

Ultimately, the turnaround from many vendors during the "Month of Apple Bugs
(http://projects.info-pull.com/moab/)" was exemplary - OmniGroup had a patch
out for OmniWeb within hours.
Part 2: ZERT

What is ZERT?

ZERT: ZERT (http://isotf.org/zert/) (the Zero-day Emergency Response Team)
is a group of engineers with extensive experience in reverse engineering
software, firmware and hardware coupled with liaisons from industry,
community and incident response groups. While ZERT works with several
Internet security operations and has liaisons to anti-virus and network
operations communities, ZERT is not affiliated with a particular vendor. The
purpose of ZERT is not to "crack" products, but rather to "uncrack" them by
averting security vulnerabilities in them before they can be widely
exploited.

Why do we need a Zero-day Emergency Response Team?

ZERT: In recent times there has been significantly more effort by criminal
elements to exploit zero-day vulnerabilities for financial gain. In some
cases the pervasiveness of vulnerabilities presents a critical risk to a
significant number of internet users. At such time it may be desirable to
some people to have an option to implement interim safeguards until a
manufacturer's patch is available. ZERT members work together as a team to
release a non-vendor patch when a so-called "0day" (zero-day) exploit
appears in the open which poses a serious risk to the public, to the
infrastructure of the internet, or both.

What type of channels do you control to look for zero-day exploits?

ZERT: ZERT members belong to and work with a wide variety of organisations
that encounter zero-day exploits or reports thereof.

At the moment you have released some patches for Microsoft software. Do you
plan to work on other vendors' software too?

ZERT: When a zero-day is announced members of ZERT will discuss the
perceived exploitability of the vulnerability and the anticipated impact. If
the vulnerability is deemed to be critical enough to warrant a patch and the
members have the time and skills to create a patch that they are comfortable
with then the name of the vendor is not really a concern.

How much time do you need to develop a patch? How much additional effort is
need to support localised versions of the software?

ZERT: The amount of time to develop, test and release a patch will vary from
vulnerability to vulnerability. There really haven't been enough patches
released by ZERT to have a meaningful historical reference. To date we have
not discussed the localisation of patches.

Generally closed-source vendors take a long time to release patches. They
often claim that this time is required to test all possible configurations
and interactions with other software to avoid breaking production systems.
How can your approach be faster even without access to source code? What
type of reliability testing do you make?

ZERT: ZERT always recommends that anyone using a third party patch perform
extensive testing in their environment to determine the suitability and
compatibility of the patch. ZERT cannot perform the in-depth testing that we
would expect of the vendor. If a person is faced with taking one or more
servers or workstations offline or trying a third party patch until the
vendor has an authorized solution, the choice may be to risk a patch that
has undergone less than optimal testing. Generally, if vendor supplied
workarounds are viable those would be a first choice.

ZERT developers and beta testers test the patches on a variety of systems,
but we do not claim to be able to perform the exhaustive testing that a
vendor would. That is why ZERT always recommends caution in the use of third
party patches and proper testing in the user's environment.

Did you have any legal problem posting some disassembled code from a
Microsoft patch in your paper [PDF
(http://zert.isotf.org/papers/vml-details-20061004.pdf)]?

ZERT: No, we have not had any legal problems at all.

ZERT has a legal advisor, but we have not had legal issues at all, other
than how we wish to license our patches. ZERT does not publish
vulnerabilities, ZERT provides patches for published vulnerabilities.

>From your analysis of official patches from Microsoft, what can you say?
What approach do they use when fixing bugs?

ZERT: ZERT has no interest in the variety of ways that a vendor may choose
to approach the problems they solve. Vendors must choose the methods they
deem most suitable on a case by case basis. ZERT's focus is generally on
interim solutions for critical vulnerabilities.

How do you install your patches? Do you need to include some pieces of the
file you are going to patch? Do you think this could become a legal problem?

ZERT: There are a variety of ways to patch vulnerabilities. Care is taken
not to redistribute code in a manner that would violate license agreements.
We have had no reason to expect any legal problems to date and do not expect
any in the future. We license our software as open source under BSD or GPL.

Will new Vista security mechanisms block the use of external patches?

ZERT: This will depend on what needs to be patched. As long as it is not the
kernel or a signed driver it is unlikely to have much effect on third party
patches, and probably virtually no effect on patching vulnerabilities for
other vendor's products.

Does Microsoft EULA limit your power to develop and/or distribute fixes?

ZERT: Microsoft EULAs do not apply to Linux, Apple, Sun, Dlink, Linksys,
etc. ZERT is not a vendor centric organisation. We have not had any issues
with Microsoft in regards to EULA infringements.

Are you aware of any new clause included in Windows Vista EULA that could
affect your work?

ZERT: ZERT enjoys open communications with Microsoft. If a Microsoft EULA is
ever an issue with respect to a ZERT patch we are confident that Microsoft
will advise us of an concerns. We do not anticipate any issues with any
vendor's EULAs however.

Federico Biancuzzi is freelancer. In addition to SecurityFocus he also
writes for ONLamp, LinuxDevCenter, and NewsForge.

This article originally appeared in Security Focus
(http://www.securityfocus.com/columnists/437/).

Copyright © 2007, SecurityFocus (http://www.securityfocus.com/)