AV-Comparatives conducts a variety of malware protection tests, to ensure that all aspects of protection are covered. We test against various types of malicious programs, not only viruses, and use large sample sizes in each test to ensure statistical relevance. We also run false-positive tests to ensure that an antivirus program does not simply detect any unknown program as malicious. Each test report contains details of the aim of the test, methods and sample sizes used. We also intend to publish an overview of our test results, with guidelines for readers to help them understand which tests may be most relevant to their own particular requirements.
We would suggest that readers should read the description of each test (by whichever testing institute) carefully and try to understand which aspects are being evaluated, and also bear in mind things like the type of malware being used in the test, the sample size, the used product versions, etc. This may help to explain why a product gets good marks on one test by a particular institute, but bad marks on a different test by the same institute; also why a product scores well in a test by one institute, but badly in an apparently similar test by a different institute.
Listed below are some noteworthy institutes that provide tests of security software. The list is not meant as an endorsement, but more as a reference to some alternative sites.
Independent Testing Organizations
There are several independent testing organizations in the world. They all do their best to provide consumers with an unbiased view and assessment of the efficacy of security solutions. The services provided are useful not only for the users of the software, but also for the manufacturers. Very often, independent testing labs are used by security software vendors as an additional, external quality assessment lab, in order to find bugs and see in which areas further improvements are needed. We advise users to have a look at a variety of tests (covering different aspects of products) and testing labs (to get a broad overview over time from various sources).
AV-Test (Germany) is an independent supplier of services in the fields of IT security and antivirus research, focusing on the detection and analysis of the latest malicious software and its use in comprehensive testing of security products. AV-Test, led by Andreas Marx, provides a variety of tests on various platforms and releases them almost every month for free on their website. Recently AV-Test has focused more on certifications than on detailed comparative tests.
MRG Effitas (UK) is an independent IT security research organisation, which focuses mainly on browser and online banking security tests, but provides from time to time also other various evaluations.
NSS Labs (USA) is an independent information security research and advisory company. NSS Labs delivers various research and expert analysis to provide useful information needed to make good security decisions. NSS Labs is known mainly for their thoughtful testing of appliances and hardware security solutions.
SE Labs (UK) is an independent testing facility that evaluates personal and business technology, including hardware and software. SE Labs (previously known as DennisTechnologyLabs) is a small testing lab lead by Simon Edwards (SE), and provides interesting tests.
Virus Bulletin (UK) focuses its efforts in three main areas: an online magazine, an annual conference and product testing (VB100). Virus Bulletin’s VB100 logo award of is de facto a product certification, as the criterion is proving whether a product can detect 100% of malware samples listed on the WildList without generating any false positives.
|ICSA Labs||West Coast Labs|
Some more information and documents about test methods, guidelines and best practices for tests can be found e.g. on the AMTSO website. All the above-mentioned testing labs follow the best industry practices, so users have some valuable sources for security software tests and reviews without having to rely on “recommendations” of sales staff, marketing departments or opinions of unknown “advisors” on forums. A further interesting article about some tests and how to interpret them can be found here.
|Comparative Testing Labs||Real-World Protection Test||Number of test cases per month||File Detection Test||Number of test cases||Behavioral Test||Performance Test||Malware Removal Test||Included vendors
|AV-Comparatives||~400 (~2000 per report)||~150000||~20|
|AV-Test||~80 (~160 per report)||~15000||~20|
|SE Labs||~50 (100 per report)||N/A||~10|
Data as of October 2016