Anniversary Report 2004-2023 20 years of AV-Comparatives

Happy Anniversary!

How AV-Comparatives began

The story of AV-Comparatives began the way it does with so many computer users, namely with a virus infection. In 1993, Andreas Clementi was hit by a computer virus: the “November 17 virus – NOV_17.855”. This awakened his interest. Andreas was not satisfied with the sometimes very contradictory tests of antivirus programs in computer magazines, and so began the intensive investigation of malware and antivirus software, which continues to this day. Later he met Peter Stelzhammer, and together, they started publishing the results of the tests at www.av-comparatives.org. Twenty years ago, AV-Comparatives started regularly releasing public results in 2004. In this report, we want to not only celebrate our anniversary but also give a short overview of the last 20 years of our history.

About AV-Comparatives

AV-Comparatives is an independent organisation that conducts comprehensive testing and evaluations of antivirus and security software. It has been active in the cybersecurity industry for over two decades and provides valuable insights into the effectiveness of various security solutions.

AV-Comparatives conducts a wide range of tests aimed to assess the capabilities of antivirus and security software in protecting against malware, viruses, phishing attacks, and other online threats across different platforms and operating systems.

The organisation’s evaluations help users, businesses, and security professionals make informed decisions when selecting antivirus and security software for their devices and networks. AV-Comparatives’ reports and findings are widely respected in the cybersecurity community and are often used as benchmarks for comparing different security products.

Headquarters

© Tourismusverband Innsbruck

Our office is situated in the Tyrolean capital of Innsbruck, a beautiful and historic city with spectacular views of the mountains.

Evolution of our Tests

AV-Comparatives has undergone a remarkable evolution and expansion since its inception two decades ago. It all commenced with a pivotal File Detection Test designed to evaluate the efficacy of antivirus solutions for Microsoft Windows. This foundational test primarily focused on detecting malware samples through offline file scans. As the landscape of cybersecurity evolved, so did our test methodologies and offerings. The test was later replaced by the Malware Protection Test, which also includes the execution of the files and takes into consideration the cloud and remediation capabilities (due to which the Malware Removal standalone test, as well as the Retrospective and Behavioural standalone tests, became obsolete).

We introduced False Alarm Tests to ensure balanced product evaluations, preventing inflated scores due to overly sensitive heuristic settings. This commitment to accuracy and fairness became a hallmark of AV-Comparatives’ testing protocols. We also began conducting Performance Tests, aimed at measuring the impact of antivirus software on system performance. This addition underscored our dedication not only to security but also to user experience.

To expand our horizons further, we introduced Real-World Protection Tests in 2009. These tests serve as a state-of-the-art test for suites’ real-world defence capabilities against active malware threats. Leveraging default settings, including on-execution protection features, these tests simulate real-world scenarios to comprehensively assess the efficacy of security products.

The dawn of the mobile era in 2010 prompted us to extend our testing to Android security products, reinforcing our commitment to safeguarding diverse digital ecosystems. Concurrently, the Anti-Phishing Test was introduced in 2011, reflecting our proactive stance against emerging threats like phishing websites. Recognising the growing significance of Mac security, we introduced the Mac Security Test in 2012, broadening our scope to cover a wider spectrum of devices and operating systems. In 2014, AV-Comparatives expanded its scope to include parental control software evaluations aimed at shielding children from inappropriate content online. This initiative resonated with our ethos of promoting safe digital environments for all users. In addition to testing the effectiveness of products, AV-Comparatives also review the features and user interfaces of security products.

Subsequent years witnessed a strategic shift towards evaluating business and enterprise security software, leading to the inception of Business Security Tests and Reviews in 2018. This move underscored our adaptability to evolving cybersecurity landscapes and our commitment to serving a diverse clientele.

The Advanced Threat Protection (ATP) Test in 2019 and the Endpoint Prevention & Response Test in 2020 exemplify our relentless pursuit of innovation and relevance. These tests simulate targeted attacks and evaluate the holistic efficacy of security solutions, including their ability to mitigate advanced threats and their total cost of ownership. The Endpoint Prevention & Response Test (EPR-Test) is a comprehensive test that provides valuable insights into the effectiveness and reliability of various EPP/EDR/XDR solutions available in the market.

Since 2022, we also offer various Offensive Security Tests, such as LSASS-Protection Tests, Process-Injection Tests, Anti-Tampering Tests, NGFW-Tests, etc. Those challenging hands-on tests aim to evaluate different product protection aspects and features in depth.

Moreover, our repertoire includes specialized tests addressing contemporary security challenges, such as IoT-Tests, VPN-Tests, Stalkerware-Tests, Data-Transmission-Tests, and more. These endeavours reflect our responsiveness to emerging threats and commitment to public interest. In addition to the above-mentioned tests, we also offer a variety of internal tests performed for vendors, publishers, and enterprises, tailored to their evaluation needs.

In tandem with our expansion, we remain cognizant of the need to streamline our testing methodologies and certification programs. Some tests have been phased out or integrated into broader evaluations to ensure relevance and efficiency without compromising the depth of analysis. As we forge ahead, AV-Comparatives remains steadfast in its mission to foster a safer digital world through rigorous testing, insightful analysis, and unwavering dedication to excellence. Our journey is a testament to our resilience, adaptability, and unwavering commitment to providing independent, high-quality cybersecurity evaluations. By choosing AV-Comparatives for their security testing needs, security software developers and their clients can gain valuable insights into the efficacy of their security solutions, ensuring the protection of their digital assets and staying ahead of emerging threats.

An overview of our tests and methodologies can be found here: https://www.av-comparatives.org/test-methods/ Each test report contains also the applied test methodology.

An overview of vendors currently holding an AV-Comparatives certification for their consumer, business, or enterprise/EDR/XDR products can be found here: https://www.av-comparatives.org/security-products-overview/

The History of our Website

Our website has evolved considerably over the years. This reflects the fact that we have increased the number and types of tests and reviews we do, and that we try to make all our reports easily accessible to the public. Before the av-comparatives.org website was unveiled to the public in 2004, we had for several years our web space hosted by the University of Innsbruck. It had no graphical interface and was only used to store the test results.

Our first public web page had a relatively simple design, with a single menu column.
Five years later, we completely overhauled the website, both graphically and in terms of navigation. The result was a bright, easy-to-read modern design, which allowed for a greater number of reports and reviews. It also enabled us to introduce polls and surveys, and to link with the increasingly popular social networking sites.
In 2013, the website was again reworked to provide for new and more comprehensive reports, such as the Proactive/Behavioural Test and the Mac Security Test/Review. Increased use of graphics helped to emphasize our most recent reports.
A refinement of this came the following year, with a carousel cycling through our latest reports, and links to our latest news stories.
In 2015 we were busy to fill the website with constant test-results and content, providing the readers with an insight of the topic and try to mirror the complexity of the tests while having the opportunity to use the information from experts first hand.
From 2019 we increased our focus on testing business and enterprise products. This was reflected in the separating of the website into a consumer and an enterprise area.

Certifications

AV-Comparatives is an ISO 9001 certified organisation. We received the certificate of TÜV Austria for our management system for the scope “Independent Tests of Anti-Virus Software”, which is recertified every year.

AV-Comparatives Meets all Requirements of EICAR Standard. The European Expert Group for IT-Security (EICAR) has, as part of its initiative EICAR Minimum Standards, certified the first IT-security testing lab.

Awards

AV-Comparatives and its test methodologies have been awarded several times in the past years.

• Constantinus Award for Computer Science and Special Award International Project – given by the Professional Association of Management Consultancy and Information Technology
• Austrian eAward – given by the Office of the Federal Chancellor
• Cluster Award – given by the Standortagentur Tirol for the best IT innovation of the year
• Innovationspreis IT – “Best Of” – given by Initiative Mittelstand Germany
• Best Paper Award at Malware Conference

Funding and Partners

As an institute that works for the common good, AV-Comparatives is supported by numerous academic and public bodies, including the following:

• The State of Austria
• The European Union
• Laura Bassi Center of Expertise, Innsbruck
• University of Innsbruck, Research Group for Databases and Information Systems
• University of Innsbruck, Research Group Quality Engineering
• University of Innsbruck (Tyrol IT Group)
• Management Center Innsbruck
• Ulysseus European University
• Hong Kong University of Science and Technology
• National Taiwan University of Science and Technology
• Brandeis University Massachusetts, Computer Science Department

Magazines & Analysts Relations

AV-Comparatives’ test results are used by nearly all major IT security magazines, also by the analysts at the research companies Gartner, Forrester, Frost & Sullivan, etc., and are often quoted by antivirus vendors. We aim to be as open as possible, and so vendors and media can visit our offices and observe tests being carried out. Various tests are regularly commissioned and/or published by specialist IT magazines around the globe.

Cooperations and Research

AV-Comparatives has successfully conducted and completed over ten diverse projects in collaboration with various universities in the past decades, generously funded by the European Union. These initiatives have spanned several key areas within the cybersecurity domain, covering OSINT research, cyber-physical systems in the automotive industry and new ways of red team testing. This significantly contributes to our deepened understanding and enhanced defensive capabilities against emerging cyber threats.

AV-Comparatives publications are used as standard reference texts at universities and serve as reference and a basis for further research.

IoT-Research

AV-Comparatives is enthusiastically entering a forward-thinking partnership with The Management Centre Innsbruck (MCI), jointly initiating a comprehensive research venture into the field of Internet of Things (IoT) security. This innovative and ambitious project has received significant support from the Austrian Government, reflecting the national commitment to advance in this critical field of cybersecurity. A substantial investment has been earmarked for five years, ensuring that our collaborative endeavour will push new boundaries in the IoT security landscape.

20-Years Continuous Participation Award

Over the last twenty years, AV-Comparatives has tested (publicly and internally) numerous security products from hundreds of security vendors. The manufacturers Avast, Avira, Bitdefender, ESET, Kaspersky, and McAfee have participated continuously in our public tests since 2004. We would like to congratulate them on their ongoing commitment to product improvement. This sometimes involves a less than perfect result but leads to better products and better protection of the users.

We are delighted to give these six vendors our 20-Years Continuous Participation Award.

Here’s what the six vendors have to say about us:

Avast: “For the past twenty years, AV-Comparatives has delivered valuable insights and transparency both for security vendors looking for comparison to their competitors, and for customers providing guidance to select the best solution for their own use. But there always has been more between vendors and the AV-Comparatives – it is the part where the testers challenge the vendors to deliver even better solutions protecting their customers. Going beyond the traditional anti-malware testing is the key to the future! And AV-Comparatives is pioneering in many areas and is continuously challenging the security industry. We look forward to working with AV-Comparatives for many more years.”

Avira: “For more than 20 years, Avira has maintained pride in subjecting its products to rigorous testing and validation by experts at AV-Comparatives, a testament to our unwavering commitment to quality and protection efficacy of our products.”

Bitdefender: “While we celebrate two decades of partnership, we recognize the remarkable evolution of AV-Comparatives, confirming their leadership and contribution to the overall cybersecurity landscape. The tests offer consumers a reliable benchmark to make informed decisions and support vendors in refining their solutions. AV-Comparatives’ rigorous methodology has not only validated Bitdefender’s excellence but also driven innovation, ensuring products meet the escalating challenges of digital threats. As we mark 20 years, this synergy remains a testament to commitment and excellence in cybersecurity, benefiting both the end-user and the industry at large, through consistent, high-quality evaluations and actionable insights.”

ESET: “In its 20 years of testing, AV-Comparatives has evolved so that its testing stayed relevant, meeting the needs of the users by following all the new technologies deployed by both security products and cybercriminals, coming up with new tests backed by state-of-the-art testing methodologies.”

Kaspersky: “We would like to congratulate AV-Comparatives on its 20^th anniversary – another decade of true independent security solutions assessment expertise. Non-biased testing remains an important driver for the security industry, and by challenging us to go further, they help Kaspersky deliver stronger cybersecurity solutions and be more competitive. We thank the team for this and for the productive feedback, and look forward to working with AV-Comparatives for many more years to come.”

McAfee: “Congratulations to all at AV-Comparatives on their 20^th anniversary! For two decades, McAfee has found immense value not only in the team’s rigorous testing, but also in their exceptional knowledge of malware, both of which are among the best in the industry. Additionally, the forum AV-Comparatives provides allows for robust conversations around testing topics that ensure we are all aligned to driving the best outcomes in software security across the industry. We look forward to many more years of partnership between McAfee and AV-Comparatives.”