AV-Comparatives conducts a variety of malware protection tests, to ensure that all aspects of protection are covered. We test against various types of malicious programs, not only viruses, and use large sample sizes in each test to ensure statistical relevance. We also run false-positive tests to ensure that an antivirus program does not simply detect any unknown program as malicious. Each test report contains details of the aim of the test, methods and sample sizes used. We also intend to publish an overview of our test results, with guidelines for readers to help them understand which tests may be most relevant to their own particular requirements.
We would suggest that readers should read the description of each test (by whichever testing institute) carefully and try to understand which aspects are being evaluated, and also bear in mind things like the type of malware being used in the test, the sample size, the used product versions, etc. This may help to explain why a product gets good marks on one test by a particular institute, but bad marks on a different test by the same institute; also why a product scores well in a test by one institute, but badly in an apparently similar test by a different institute.
Listed below are some noteworthy institutes that provide tests of security software. The list is not meant as an endorsement, but more as a reference to some alternative sites.
Independent Testing Organizations
There are several independent testing organizations in the world. They all do their best to provide consumers with an unbiased view and assessment of the efficacy of security solutions. The services provided are useful not only for the users of the software, but also for the manufacturers. Very often, independent testing labs are used by security software vendors as an additional, external quality assessment lab, in order to find bugs and see in which areas further improvements are needed. We advise users to have a look at a variety of tests (covering different aspects of products) and testing labs (to get a broad overview over time from various sources).
AV-Test (Germany) is an independent supplier of services in the fields of IT security and antivirus research, focusing on the detection and analysis of the latest malicious software and its use in comprehensive testing of security products. AV-Test, led by Andreas Marx, provides a variety of tests on various platforms and releases them almost every month for free on their website. Recently AV-Test has focused more on certifications than on detailed comparative tests.
Dennis Technology Labs
Dennis Technology Labs (UK) is an independent testing facility that evaluates personal and business technology, including hardware and software. Dennis Technology Labs (aka DTL) is a small testing lab from Dennis Publishing, and provides interesting tests.
MRG Effitas (UK) is an independent IT security research organisation, which focuses mainly on browser and online banking security tests, but provides from time to time also other various evaluations.
NSS Labs (USA) is an independent information security research and advisory company. NSS Labs delivers various research and expert analysis to provide useful information needed to make good security decisions. NSS Labs is known mainly for their thoughtful testing of appliances and hardware security solutions.
PC Security Labs
PC Security Labs (China) is an independent Chinese testing agency providing various security software evaluations. PC Security Labs (aka PCSL) is currently mainly focussing on Mobile (Android) Security Software Tests.
Virus Bulletin (UK) focuses its efforts in three main areas: a monthly magazine, an annual conference and a bimonthly product certification (VB100). Virus Bulletin’s VB100 logo award of is de facto a product certification, as the criterion is proving whether a product can detect 100% of malware samples listed on the WildList without generating any false positives. The website only lists Pass/Fail for each tested product. Current up-to-date comparative reports including detailed results and comments are only available for a fee (~20 USD).
|ICSA Labs||West Coast Labs|
Some more information and documents about test methods, guidelines and best practices for tests can be found e.g. on the AMTSO website. All the above-mentioned testing labs follow the best industry practices, so users have some valuable sources for security software tests and reviews without having to rely on “recommendations” of sales staff, marketing departments or opinions of unknown “advisors” on forums. A further interesting article about some tests and how to interpret them can be found here.
|Comparative Testing Labs||Real-World Protection Test||Number of test cases per month||File Detection Test||Number of test cases||Behavioral Test||Performance Test||Malware Removal Test||Included vendors
|AV-Comparatives||~1000 (~4000 per report)||~125000||~25|
|AV-Test||~100 (~200 per report)||~20000||~25|
|Dennis Technology Labs||~50 (100 per report)||N/A||~10|
|PC Security Labs||N/A||~20000||~25|
Data as of May 2014