Computer Magazines and Virus Testing
Courtesy of the alt.comp.virus newsgroup participants.
(These "anti-malware" pages are the result of a continuing cooperative effort.)
by Andrew J. Lee
AVIEN Founding Member
It is indisputable that any magazine can test and compare the usability,
the interface, the update method, the system performance impact, the "user
friendliness" and the features of respective products, and, on that basis,
many magazines have conducted good and fair reviews of the anti-virus
However, on the basis of their stated methodology for testing the virus
detection functionality of the scanners, they often have not.
The idea that a magazine will be able to test any virus scanner with their
own "quarantined" virus collection is at best foolish and at worst dangerous.
Let me put it simply.
When it comes to Scanner testing such magazines usually do not know what
they are doing.
This is proved by telling us how their test was conducted.
It is simply wrong to assume that they can test a scanner just by seeing if
it detects the viruses that they have.
If it detects them they have proved nothing, except that there are some
files they suspect of being viruses that it detects, you cannot extrapolate
any further conclusion. If it does not detect, they have no way of telling
This is because they don't know whether their samples are viable* either
fully or in part, nor whether the samples they have are mutations or
variants (i.e. someone or something has made changes to it).
The major criticisms that I have of such methodologies are these:
Therefore such tests have proved nothing, and are of little value in making
a purchasing judgement.
- They do not define and publish the sample set used - listing by family,
variant and type.
- They have not tested the ability to replicate, (the definition of a
virus), of each member of that sample set.
- They do not publish the methodology of testing, which must be
consistent for each product, i.e. how they set it up, were the files tested
against in their natural state (as they would appear in the wild) etc.
- They do not state whether they have distinguished viruses from Trojans
or other non viral malware.
- They often state disinfection or healing as a benefit, when it is far
from agreed that it is of any benefit.
- They often do not state the update or engine level of each product, nor
the platforms on which they tested.
For reliable results check the tests done by respected independent bodies
in the field, you will often see that their testing contradicts such
arbitrary magazine test results.
See these links for some real tests :
Real world anti-virus scanner testing is carried out using thousands of
verified viruses under strictly controlled conditions. They are also
carried out, at least the recognized tests, by experts in the field, who
understand not only the implications of the results, but who are able to
correctly interpret the results.
Any tests a computer magazine have conducted in the manner described
earlier are immediately invalidated by the non scientific method.
*Viable here means able to replicate and infect other files.
Outraged of Slovakia - Virus Bulletin
NOD32 trashed by CNet / ZDNet review !!!
© Claymania Creations 2001 - 2013. All rights reserved.
Updated: May 29, 2002