AV-Comparatives weblog

If you want to follow us on Facebook, click here and add yourself as Fan!

We will be at the CeBIT 2010 in Hannover (Germany) from 2nd to 6th of March 2010
You can find us at the CeBIT Security Plaza, Hall 11.

If you want to meet us, please arrange a meeting. Just mail us.

Hi,

Andreas will be at Virusbulletin all days. If you want to meet Peter he will be available for arranged meetings on the 24th of September. Just do an appointment per mail.

Thanks and have a good conference!

We had a problem with our contact form on the website, due which we could not get messages sent between the begin of May and the 7th June. If you tried to contact us in that period and have not received any reply from us, please resend your message using the new contact form (http://www.av-comparatives.org/contact ). Thanks!

During the next weeks there will be some changes to our website:

• redesigned logo
• redesigned awards
• redesign/relaunch of the website
• translations in several languages
• etc.

To stay informed just subscribe to our newsletter at http://www.av-comparatives.info

We have just released various test reports translated into Spanish. You can find them at the bottom of the Comparatives section of our website.

Peter and I are going to the next AMTSO* meeting. We will return on Sunday – all emails will be answered on return.

* AMTSO (The Anti-Malware Testing Standards Organization ) is an international non-profit association that focuses on the addressing the global need for improvement in the objectivity, quality and relevance of anti-malware testing methodologies.

P.S.: last weeks we were and still are busy with some new test (beside the retrospective test which has also to be done). I guess we will release it in some weeks, so check our site for news from time to time.

I (ac) just came back to IBK from Ottawa. The VB conference is IMO the best of all, mainly because almost everyone goes there, and after all, meeting peoples is the most interesting part. I enjoyed to meet and talk with peoples e.g. from (in no specific order): K7 Computing, Kingsoft, WildList, ESET, Sunbelt Software, AVG Technologies, TrustPort, NSS Labs, VirusBuster, Symantec, Avira, Sophos, KasperskyLab, VirusBulletin, Alwil Software, G DATA Software, McAfee, F-Secure, BitDefender, Microsoft, MessageLabs, etc. (sorry if I forgot someone).

Two presentations I remember and I liked most were:

- Rebuilding testing for the future
- The AV industry: Quo Vadis?

I guess during the next weeks the presentations (as well as pictures from the conference) will be available on the VirusBulletin website, so keep watching their site.

Next year the VirusBulletin conference will be in Switzerland – I will be there. And you?

(ac)

P.S.: my mailbox exploded in my absence, so if you got the message ~”mailbox is full” or you do not get a reply within this week, please resend your mail.

The updated August 2008 test report has now been upload. Some minor changes were applied, as well as the noncompetitive iclusion of the enterprise version of McAfee (with Artemis); the detailed report about Artemis (and some other reports) will be published in some weeks.

Please download this updated report from our website.

AV-Comparatives released an updated document containing the FAQ’s and methodology. You can find the PDF document on this page.

Some weeks ago we started moving office and now the new PC’s etc. are at the new location. We already started the August test some days ago, but due some unexpected required upgrades at the infrastructure (e.g. intensity of amperage, air cooling system, etc.) and the expanded tests (the August test will include also a false alarm test etc.), we will probably delay the release of the report to around mid September (instead the 1st September), as due the upgrades the tests need to be partially interrupted for some days.

available here (simple and traditional chinese):

http://www.av-comparatives.org/seiten/ergebnisse/report18chinese.pdf
http://www.av-comparatives.org/seiten/ergebnisse/report18sc.pdf

(translated by a third party = we do not take any responsability in case of wrong translation)

The QA report of the August 2007 test set has been released. It can be found on the website in the Comparatives section.

As announced publicly several times already last year, AV-Comparatives provides since 2008 its services for a fee. The fee is NOT for the tests itself, but for the various services we provide (bug reports, usage of our material/logos for reprint and marketing material, getting sample lists, getting false alarms after the public tests, internal services, etc.). Vendors are not obligated to pay for the FULL-service-package, and if a vendor e.g. prefers to do not get all services, the fee is lower. All major testers get paid for the services they provide, this is a common practice. Of course, the fee has no influence at all on the results and is to cover our expenses and time involved.
For users and magazines the public results are free of charge, as long the source is given. If we have time and e.g. a magazine wants us to make an additional work/test of e.g. products that are not already tested, this costs some money which has to be covered by the magazine (only for the service provided to the magazine; the vendor of the tested product will not get any service).

Some pictures from the CeBIT 2008 can be found in our forum

There is currently a third party survey on our website until the end of March. Those that participate in the survey have the chance to win an 1GB USB pen drive.

I will be away the first week of March. Test reports etc. are currently scheduled for release on Monday, 10th March, ~2 PM pacific time. All non-vendors emails will be answered on my return.

For those that did not already read this info on our website:
Starting from 2008, AV-Comparatives will – like most other testers do – no longer provide its services for free, as the expenses for the site and all the work involved are too high. The vendors will not pay for the big tests itselfs (and it has of course no influence of any kind on the test results or award system) – the participation fee includes e.g. the usage of the reached award for marketing purposes and other services (like getting false positives and missed malware after the tests, bug reports, etc.). The number of participants will be limited to about 18 vendors.

The independent and well-known security software tester Mr. Marx from AV-Test.org presented at the VirusBulletin conference a very interesting paper (full paper will be released soon on the AV-Test.org website) about the current desolate state of the WildList and suggestions about how to improve it. In 1999 Dr. Bontchev also showed many problems that the WildList has; you can read about it here. Already at the AV Testing Workshop in Rekjavik 2007 most of the technical staff of the AV vendors admitted that the WildList is well-accepted and loved because it is easy to pass tests based on the WildList and because it is good for the marketing (100% detection*). So you may ask, why – if it is easy to pass – some vendors fail at detecting all samples from the WildList? The reasons could be either errors by the testers, temporary bugs in the software, but more often it is because a) more variables than just detecting all samples are needed to pass (e.g. no false positives in case of VB100), b) sometimes also very old threats that were on the wildlist 10 years ago (e.g. boot sector viruses) are still included, and probably also because not all vendors receive the WildCore collection and therefore are not tested under same circumstances. Anyway, as Dr. Bontchev pointed, the samples on the WildList are not viruses that are really out there and the malware which is really out there is not on the WildList. So, who wants to keep the WildList alive? Of course (beside marketing** peoples and certification bodies which get lot of money for quite easy to do [and for av vendors to pass paid] tests) all those vendors that know that their product would not score well in tests using large test-sets or if the WildList would get improved by adding more threats that are really out there. One vendor (which I will not name) wrote in a blog an entry basically defending the WildList and saying more or less that tests based on large sets are not as accurate and that perhaps such testers rely on other AV scanner to verify if something is malicious (which I can assure you that neither Mr. Marx, nor we, do that – some AV vendors do that, but that’s another story…). One could, perhaps, think that maybe the above vendor is esp. interested in defending tests based on the WildList, because one company which does such tests is (at least currently still) their sister company.

The presentation of Mr. Marx was IMO one of the top 5 at the VB conference. Let’s hope that some of the suggestions on how to improve the WildList in future can be addressed and applied, otherwise let’s hope that the WildList finally really dies.

* based on the WildList xx/200x (most buyers reading this on the box do not understand what it means, but 100% sounds good)

** reading on a box e.g. “detects 90%” or “detects 98%” of malicious software does of course not sound as good and reliable as “detects 100%”

P.S.: AV-Comparatives does not provide tests based on the WildList.

I am NOT saying that ITW tests are completely useless, nor am I saying that tests based on large test-sets give you the best insight. The loyal readers of AV-Comparatives know what we always state: do not look/rely just on one single test from one testing site only, do not rely on test results alone, but look at the bigger picture and consult as many independent tests and reviews you can find to build up an opinion and get than your own opinion later by trying out the various products by yourself on your system.