avc-community Welcome, AV-Comparatives partners and friends!

This is a place of sharing and exchanging information, the hub of the AV-Comparatives community, where we keep in touch with our partners and friends. We’d like to thank you all for your past, present and future support and for helping us emphasize the importance of independent testing of security software.

List of AV Testing Labs

AV-Comparatives conducts a variety of malware protection tests, to ensure that all aspects of protection are covered. We test against various types of malicious programs, not only viruses, and use large sample sizes in each test to ensure statistical relevance. We also run false-positive tests to ensure that an antivirus program does not simply detect any unknown program as malicious. Each test report contains details of the aim of the test, methods and sample sizes used. We also intend to publish an overview of our test results, with guidelines for readers to help them understand which tests may be most relevant to their own particular requirements.
We would suggest that readers should read the description of each test (by whichever testing institute) carefully and try to understand which aspects are being evaluated, and also bear in mind things like the type of malware being used in the test, the sample size, the used product versions, etc. This may help to explain why a product gets good marks on one test by a particular institute, but bad marks on a different test by the same institute;  also why a product scores well in a test by one institute, but badly in an apparently similar test by a different institute.
Listed below are some noteworthy institutes that provide tests of security software. The list is not meant as an endorsement, but more as a reference to some alternative sites.

Independent Testing Organizations
There are several independent testing organizations in the world. They all do their best to provide consumers with an unbiased view and assessment of the efficacy of security solutions. The services provided are useful not only for the users of the software, but also for the manufacturers. Very often, independent testing labs are used by security software vendors as an additional, external quality assessment lab, in order to find bugs and see in which areas further improvements are needed. We advise users to have a look at a variety of tests (covering different aspects of products) and testing labs (to get a broad overview over time from various sources).

AV-Comparatives

AV-Comparatives (Austria) is an independent security software testing organization, which provides a multitude of top-quality and state-of-the-art tests, like the “Whole Product Real World Dynamic Protection Test”, which evaluates the protection provided by the security software as a whole. The detailed test reports are provided for free to the public. AV-Comparatives aims to provide valuable and unbiased information to consumers (such as this list of further testing labs), as well as some free public services like the Android APK Analyser (AVC UnDroid).

av com

AV-Test

AV-Test

AV-Test (Germany) is an independent supplier of services in the fields of IT security and antivirus research, focusing on the detection and analysis of the latest malicious software and its use in comprehensive testing of security products.
AV-Test, led by Andreas Marx, provides a variety of tests on various platforms and releases them almost every month for free on their website. Recently AV-Test has focused more on certifications than on detailed comparative tests.

av test

Dennis Technology Labs

Dennis Technology Labs

Dennis Technology Labs (UK) is an independent testing facility that evaluates personal and business technology, including hardware and software.
Dennis Technology Labs (aka DTL) is a small testing lab from Dennis Publishing, and provides interesting tests.

den tech

MRG Effitas

MRG Effitas

MRG Effitas (UK) is an independent IT security research organisation,
which focuses mainly on browser and online banking security tests, but provides from time to time also other various evaluations.

MRG

NSS Labs

NSS Labs

NSS Labs (USA) is an independent information security research and advisory company. NSS Labs delivers various research and expert analysis to provide useful information needed to make good security decisions. NSS Labs is known mainly for their thoughtful testing of appliances and hardware security solutions.

nss labs

PC Security Labs

PC Security Labs

PC Security Labs (China) is an independent Chinese testing agency providing various security software evaluations.
PC Security Labs (aka PCSL) is currently mainly focussing on Mobile (Android) Security Software Tests.

pc sec

Virus Bulletin

Virus Bulletin

Virus Bulletin (UK) focuses its efforts in three main areas: a monthly magazine, an annual conference and a bimonthly product certification (VB100). Virus Bulletin’s VB100 logo award of is de facto a product certification, as the criterion is
proving whether a product can detect 100% of malware samples listed on the WildList without generating any false positives. The website only lists Pass/Fail for each tested product. Current up-to-date comparative reports including detailed results and comments are only available for a fee (~20 USD).

virus bull

Certification Labs

Certification labs are not really intended to provide comparative test results. They exist to certify whether a product has reached a given standard. It does not usually tell the public how many trials/cycles a particular vendor’s product had to go through to reach the standard. Certifications are very useful for establishing whether a vendor is reliable (has enough money to be in the business and get certified) and if its product is able to reach a minimum standard (usually covering all files included in the WildList) at one point in time. For users, the given information is usually too limited to be useful, but for companies, certifications are very meaningful and necessary. A good paper on why product certification matters can be found here. The most well-known certification labs are ICSA Labs and West Coast Labs (both located in USA).

 

ICSA Labs West Coast Labs
icsa labs west labs

Some more information and documents about test methods, guidelines and best practices for tests can be found e.g. on the AMTSO website. All the above-mentioned testing labs follow the best industry practices, so users have some valuable sources for security software tests and reviews without having to rely on “recommendations” of sales staff, marketing departments or opinions of unknown “advisors” on forums. A further interesting article about some tests and how to interpret them can be found here.

Comparative Testing Labs Real-World Protection Test Number of test cases per month File Detection Test Number of test cases Behavioral Test Performance Test Malware Removal Test Included products
AV-Comparatives ~500 (~2000 per report) ~150000 ~25
AV-Test ~50 (~100 per report) ~20000 ~25
Dennis Technology Labs ~30 (100 per report) N/A ~10
PC Security Labs N/A ~20000 ~25
VirusBulletin N/A ~3000 ~50

Data as of 10th of December 2013

About AV-Comparatives

AV-Comparatives is an independent not-for-profit organization offering systematic testing that checks whether security software, such as PC/Mac-based antivirus products and mobile security solutions, lives up to its promises. Using one of the largest sample collections worldwide, it creates a real-world environment for truly accurate testing. AV-Comparatives offers freely accessible results to individuals, news organizations and scientific institutions. Certification by AV-Comparatives provides an official seal of approval for software performance which is globally recognized. Currently, AV-Comparatives' Real-World Protection Test is the most comprehensive and complex test available when it comes to evaluating the real-life protection capabilities of antivirus software. Put simply, the test framework replicates the scenario of an everyday user in an everyday online environment – the typical situation that most of us experience when using a computer with an Internet connection. AV-Comparatives works closely with several academic institutions, especially the University of Innsbruck’s Department of Computer Science, to provide innovative scientific testing methods.
If you plan to buy an Anti-Virus, please visit the vendor's site and evaluate their software by downloading a trial version, as there are also many other features and important things for an Anti-Virus that you should evaluate by yourself. Even if quite important, the data provided in the test reports on this site are just some aspects that you should consider when buying Anti-Virus software.

AVC Analyzer

undroid