[Infowarrior] - AV vendors to agree on standard testing guidelines

[Richard Forno]

Rating antivirus software: vendors to agree on standard testing guidelines

By Joel Hruska | Published: December 05, 2007 - 01:00AM CT

http://arstechnica.com/news.ars/post/20071205-antivirus-vendors-to-agree-on-
testing-guidelines.html

Understanding which AV package provides the best level of total protection
isn't easy. Products from Symantec (Norton Antivirus) and McAfee are
virtually ubiquitous, but there are a dozen or more smaller players in the
market, all of which advertise themselves as being the best solution for
total antivirus protection. Since each company creates its own benchmarks
and comparisons, though, it's virtually impossible for an end user to
compare one product against another. AV manufacturers are aware of this
problem, and are working collectively towards a solution. As PC World
reports, many of the larger players in the AV market met in Seoul last week
to form the Anti-Malware Testing Working Group. The new group will be tasked
with creating a set of software benchmarks that can conduct behavioral tests
on multiple suites of security software.

Currently, most comparative AV tests are signature-based. This type of test
is analogous to what occurs when an antivirus product runs a hard drive
scan‹virus files with various signatures are scattered throughout the data
set that's being checked and each product is rated on how many of those
various files it managed to detect. Behavioral scans, on the other hand, are
meant to replicate how a PC typically encounters malware, and they model a
wide variety of scenarios from email virus detection to page redirects.

Companies that have signed on to work with the new group include Symantec,
F-Secure, and Sunbelt Software (no McAfee yet). The AMTWG isn't just a
vendors-only club; AV software evaluators are also participating, including
AV-Test.org and Virus Bulletin. Although they aren't listed in the PC World
article, I contacted AV-Comparatives.org. The company publishes a quarterly
"report card" on AV products, and is considered to be an excellent evaluator
of security products.

AV-Comparatives tells Ars that it has also been involved in some of these
vendor meetings, and reports that the various companies involved seem
genuinely interested in creating an effective standard for measuring product
performance. Vendors also aren't putting any pressure on product testers to
specifically adopt or abandon particular methods of testing, and have thus
far emphasized that the goal of the task force is to create a good
evaluation tool.

No single benchmark is ever perfect‹even those constructed for comprehensive
testing and created with the best of intentions are only capable of modeling
a certain number of scenarios‹but the AV industry appears to be taking the
first step towards providing customers with an actual metric they can use
for comparative purposes.