The independent and well-known security software tester Mr. Marx from AV-Test.org presented at the VirusBulletin conference a very interesting paper (full paper will be released soon on the AV-Test.org website) about the current desolate state of the WildList and suggestions about how to improve it. In 1999 Dr. Bontchev also showed many problems that the WildList has; you can read about it here. Already at the AV Testing Workshop in Rekjavik 2007 most of the technical staff of the AV vendors admitted that the WildList is well-accepted and loved because it is easy to pass tests based on the WildList and because it is good for the marketing (100% detection*). So you may ask, why – if it is easy to pass – some vendors fail at detecting all samples from the WildList? The reasons could be either errors by the testers, temporary bugs in the software, but more often it is because a) more variables than just detecting all samples are needed to pass (e.g. no false positives in case of VB100), b) sometimes also very old threats that were on the wildlist 10 years ago (e.g. boot sector viruses) are still included, and probably also because not all vendors receive the WildCore collection and therefore are not tested under same circumstances. Anyway, as Dr. Bontchev pointed, the samples on the WildList are not viruses that are really out there and the malware which is really out there is not on the WildList. So, who wants to keep the WildList alive? Of course (beside marketing** peoples and certification bodies which get lot of money for quite easy to do [and for av vendors to pass paid] tests) all those vendors that know that their product would not score well in tests using large test-sets or if the WildList would get improved by adding more threats that are really out there. One vendor (which I will not name) wrote in a blog an entry basically defending the WildList and saying more or less that tests based on large sets are not as accurate and that perhaps such testers rely on other AV scanner to verify if something is malicious (which I can assure you that neither Mr. Marx, nor we, do that – some AV vendors do that, but that’s another story…). One could, perhaps, think that maybe the above vendor is esp. interested in defending tests based on the WildList, because one company which does such tests is (at least currently still) their sister company.
The presentation of Mr. Marx was IMO one of the top 5 at the VB conference. Let’s hope that some of the suggestions on how to improve the WildList in future can be addressed and applied, otherwise let’s hope that the WildList finally really dies.
* based on the WildList xx/200x (most buyers reading this on the box do not understand what it means, but 100% sounds good)
** reading on a box e.g. “detects 90%” or “detects 98%” of malicious software does of course not sound as good and reliable as “detects 100%”
P.S.: AV-Comparatives does not provide tests based on the WildList.
I am NOT saying that ITW tests are completely useless, nor am I saying that tests based on large test-sets give you the best insight. The loyal readers of AV-Comparatives know what we always state: do not look/rely just on one single test from one testing site only, do not rely on test results alone, but look at the bigger picture and consult as many independent tests and reviews you can find to build up an opinion and get than your own opinion later by trying out the various products by yourself on your system.