Computer Magazines and Virus Testing

Courtesy of the alt.comp.virus newsgroup participants.
(These "anti-malware" pages are the result of a continuing cooperative effort.)


Main Menu

by Andrew J. Lee
AVIEN Founding Member
http://www.avien.net

It is indisputable that any magazine can test and compare the usability, the interface, the update method, the system performance impact, the "user friendliness" and the features of respective products, and, on that basis, many magazines have conducted good and fair reviews of the anti-virus software included.

However, on the basis of their stated methodology for testing the virus detection functionality of the scanners, they often have not. The idea that a magazine will be able to test any virus scanner with their own "quarantined" virus collection is at best foolish and at worst dangerous.

Let me put it simply. When it comes to Scanner testing such magazines usually do not know what they are doing. This is proved by telling us how their test was conducted. It is simply wrong to assume that they can test a scanner just by seeing if it detects the viruses that they have. If it detects them they have proved nothing, except that there are some files they suspect of being viruses that it detects, you cannot extrapolate any further conclusion. If it does not detect, they have no way of telling why.

This is because they don't know whether their samples are viable* either fully or in part, nor whether the samples they have are mutations or variants (i.e. someone or something has made changes to it). The major criticisms that I have of such methodologies are these:

1. They do not define and publish the sample set used - listing by family, variant and type.
2. They have not tested the ability to replicate, (the definition of a virus), of each member of that sample set.
3. They do not publish the methodology of testing, which must be consistent for each product, i.e. how they set it up, were the files tested against in their natural state (as they would appear in the wild) etc.
4. They do not state whether they have distinguished viruses from Trojans or other non viral malware.
5. They often state disinfection or healing as a benefit, when it is far from agreed that it is of any benefit.
6. They often do not state the update or engine level of each product, nor the platforms on which they tested.

Therefore such tests have proved nothing, and are of little value in making a purchasing judgement.

For reliable results check the tests done by respected independent bodies in the field, you will often see that their testing contradicts such arbitrary magazine test results. See these links for some real tests :

http://www.av-test.org/index.php3?lang=en
http://www.virusbtn.com/100
http://agn-www.informatik.uni-hamburg.de/vtc/
ftp://agn-www.informatik.uni-hamburg.de/pub/texts/tests/pc-av/2001-07/0xecsum.txt
http://www.uta.fi/laitokset/virus/
http://www.check-mark.com/cgi-bin/redirect.pl
http://www.icsalabs.com/html/communities/antivirus/certifiedproducts.shtml

Real world anti-virus scanner testing is carried out using thousands of verified viruses under strictly controlled conditions. They are also carried out, at least the recognized tests, by experts in the field, who understand not only the implications of the results, but who are able to correctly interpret the results. Any tests a computer magazine have conducted in the manner described earlier are immediately invalidated by the non scientific method.

*Viable here means able to replicate and infect other files.


Read more...

Open Letter

Outraged of Slovakia - Virus Bulletin

NOD32 trashed by CNet / ZDNet review !!!