Thursday, October 29th 2009, 9:07pm
Where can I find AMTSO compliant tests?
Quoted
We sometimes get this question, so I try to answer it here. From email conversations it seems as though a misunderstanding has arisen about what AMTSO currently does and doesn’t do due some posts by an AMTSO member who sadly has never showed up so far at any AMTSO meeting, and has therefore misunderstood some things (or had other expectations).
AMTSO has a document with 9 “fundamental principles of testing”. You can find it on their website. Basically, a test needs to follow those fundamental principles to be "compliant" (though that term has no formal definition at the moment). This means that ANY kind of test, as long as it follows those 9 principles, can be considered compliant. You will find many testers that already follow those principles. And many tests which do not. Therefore, I advise readers to check out the principles and check if a test is in their own eyes compliant or not.
AMTSO does not consist only of vendors, testers (including AV-Comparatives), reviewers and publishers are also active and founding members of AMTSO.AMTSO has also a Review Analysis Board (RAB). Should a test claim to be AMTSO compliant or have a big media impact and someone feels that the test is not accurate (i.e. not following the fundamental principles of testing), they can submit it for review to the RAB. The Board will then check the test against the 9 principles and state whether the test is "compliant" or not. As the Board consists of AV vendor researchers, testers etc. who also have other things to do, it is not reasonable to expect that it will review ALL tests out there, and even less that it will review tests before they get released. This means that you will never see a test that can accurately be described as "AMTSO compliant" in advance.
Are dynamic tests AMTSO compliant? If they follow the fundamental principles, yes. Otherwise, no. AMTSO has released some papers about various types of testing, like dynamic testing, in-the-cloud testing, etc., which include some guidelines for reviewers and testers, as well as some things to keep in mind when performing such tests. Those papers provide currently some advice how things could be done.
Do AV-Comparatives provide dynamic tests? Yes, but so far they have only been internal tests. The first public comparative test is expected to be released in some weeks. Moreover, for next year we are already building new systems for dynamic testing.
Will my product XY be tested by AV-Comparatives in a dynamic test that is AMTSO-compliant? Please read above what AMTSO compliant currently means. Dynamic testing is expensive (esp. if intended for comparing products), but if vendor XY wishes, it can apply for a single product test at any time. In a single product test the vendor can also specify what test method it would like (what should be tested, etc.). Of course products that give inappropriate warnings when executing clean files also will get a negative verdict in single product reviews, too.
More explanations about what AMTSO compliance actually means can be read here .
The nine fundamental principles are:
- Testing must not endanger the public.
- Testing must be unbiased.
- Testing should be reasonably open and transparent.
- The effectiveness and performance of anti-malware products must be measured in a balanced way.
- Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
- Testing methodology must be consistent with the testing purpose.
- The conclusions of a test must be based on the test results.
- Test results should be statistically valid.
- Vendors, testers and publishers must have an active contact point for testing-related correspondence.
AV-Comparatives encourages readers to check themselves if tests were done following the above 9 principles or not. (Should you ever come across a test of AV-Comparatives, which in your opinion does not, please send us an email and let us know!)
