Retrospective / Proactive Test February 2011

Introduction

Anti-Virus products often claim to have high proactive detection capabilities far higher than those reached in this test. This is not just a self-promotional statement; it is possible that products reach the stated percentages, but this depends on the duration of the test-period, the size of the sample set and the used samples. The data shows how good the proactive detection capabilities of the scan-ners were in detecting the new threats used in this test. Users should not be afraid if products have, in a retrospective test, low percentages. If the anti-virus software is always kept up-to-date, it will be able to detect more samples. For understanding how the detection rates of the Anti-Virus products look with updated signatures and programs, have a look at our regular on-demand detection tests. Only the heuristic/generic detection capability was tested (offline). Some products may be had the ability to detect some samples e.g. on-execution or by other monitoring tools, like behaviour-blocker, reputation/cloud heuristics, etc. Those kinds of additional protection technologies are considered by AV-Comparatives in e.g. whole-product dynamic tests, but are outside the scope of this retrospective test. For further details please refer to the methodology documents as well as the information provided on our website.

This test report is the second part of the February 2011 test. The report is delivered in late May due the high-required work, deeper analysis and preparation of the retrospective test-set. Many new viruses and other types of malware appear every day, this is why it’s important that Anti-Virus products not only provide new updates, as often and as fast as possible, but also that they are able to detect such threats in advance (also without executing them or while offline) with generic and/or heuristic techniques. Even if nowadays most Anti-Virus products provide daily, hourly or cloud updates, without heuristic/generic methods there is always a time-frame where the user is not reliably protected.

The products used the same updates and signatures they had the 22^nd February 2011 and the same detection settings as used in February. This test shows the proactive file detection capabilities that the products had at that time. We used 1,463 new malware appeared between the 23rd February and 3rd March 2011. The following products were tested:

Tested Products

• Avira Antivirus Pro 10.0
• Bitdefender Antivirus+ 2011
• eScan Anti-Virus 11.0
• ESET NOD32 Antivirus 4.2
• F-Secure Anti-Virus 2011
• G DATA AntiVirus 2011
• Kaspersky Anti-Virus 2011
• Microsoft Security Essentials 2.0
• Panda Antivirus Pro 2011
• Qihoo 360 Antivirus 1.1
• Sophos Anti-Virus 9.5
• Trustport Antivirus 2011

Test Procedure

AV-Comparatives prefers to test with default settings. In order to get comparable results we set also the few remaining products to highest settings (or leave them to lower settings) in accordance with the respective vendors. We hope that all vendors willfind the appropriate balance of detection/false alarms/system impact and will provide highest security already by default and remove paranoid set-tings inside the user interface which are too high to be ever of any benefit for normal users.

This time we tried to include in the retrospective test-set only malware which has been seen in-the-field and prevalent around the last week of February. About ¼ of the set is considered by us as “very prevalent”. As malware which became prevalentmay be spotted faster by reactive measures when many users got infected, initial proactive rates may be lower (because if they would have been spot-ted proactively, they would not become prevalent as they would be blocked in advance).

Testcases

We included in the retrospective test-set only new malware that was very prevalent in-the-field shortly after the freezing date. Samples which were not detected by the heuristic/generic detection capabilities of the products were then executed in order to see if behaviour-blocking features would stop them. In several cases, we observed that behaviour blockers only warnedabout some dropped malware components or system changes, without protecting against all the malicious actions performed by the malware; such cases were not counted as a block. As behaviour blockers only come into play after the malware is executed, a certain risk of being compromised remains (even when the security product claims to have blocked/removed the threat). Therefore, it is preferable that malware be detected before it is executed, by e.g. the on-access scanner using heuristics. This is why behaviour blockers should be considered a complement to the other features of a security product (multi-layer protection), and not a replacement.

Ranking System

The awards are given by the testers after consulting a number of statistical methods, including hierarchical clustering. We based our decisions on the following scheme:

	Proactive Protection Rates
	Under 50%	3	2	1
None - Few FPs	TESTED	STANDARD	ADVANCED	ADVANCED+
Many FPs	TESTED	TESTED	STANDARD	ADVANCED
Very many FPs	TESTED	TESTED	TESTED	STANDARD
Crazy many FPs	TESTED	TESTED	TESTED	TESTED

Test Results

The results show the proactive (generic/heuristic) file detection capabilities of the scan engines against new malware. The percentages are rounded to the nearest whole number. Do not take the results as an absolute assessment of quality – they just give an idea of who detected more, and who less, in this specific test. To know how these anti-virus products perform with updated signatures, please have a look at our on-demand tests of February and August. To know about protection rates provided by the various products, please have a look to our on-going Whole-Product Dynamic tests. Readers should look at the results and build an opinion based on their needs. All the tested products are already selected from a group of very good scanners and if used correctly and kept up-to-date, users can feel safe with any of them.

False Positive (False Alarm) Test Result

To better evaluate the quality of the detection capabilities, the false alarm rate has to be taken into account too. A false alarm (or false positive) is when an Anti-Virus product flags an innocent file to be infected when it is not. False alarms can sometimes cause as much troubles like a real infection.

The false alarm test results were already included in the test report of February. For details, please read the False Alarm Test February 2011.

1.	Microsoft	1	very few FPs (0-3)
2.	Bitdefender, eScan, F-Secure	3
3.	Sophos	4	few FPs (4-15)
4.	Avira	9
5.	Kaspersky, Trustport	12
6.	G DATA, Panda	18	many FPs (over 15)
7.	ESET	20
8.	Qihoo	104	very many FPs (over 100)

Summary Result

The below table shows the proactive on-demand detection capabilities of the various products, sorted by detection rate. The given awards are based not only on the detection rates over the new malware, but also considering the false alarm rates.

Below you can see the proactive protection results over our set of new and prevalent malware files/families appeared in-the-field (9,177 malware samples):

	Blocked	Compromised	Proactive / Protection Rate	False Alarms	Cluster
AVIRA	5414	3763	59%	few	1
Kaspersky	5047	4130	55%	few	1
G DATA	5598	3579	61%	many	2
ESET	5414	3763	59%	many	2
Panda	4772	4405	52%	many	2
Trustport	3487	5690	38%	few	2
Microsoft	3304	5873	36%	very few	2
F-secure, Bitdefender	3212	5965	35%	very few	2
eScan	3120	6057	34%	very few	2
Sophos	2111	7066	23%	few	3
Qihoo	3120	6057	34%	very many	–

Notes

Avast, AVG, K7, McAfee, PC Tools, Symantec, Trend Micro and Webroot decided to not get included in this report and to renounce to get awarded

Almost all products run nowadays by default with highest protection settings (at least either at the entry points, during whole computer on-demand scans or scheduled scans) or switch automatically to highest settings in case of a detected infection. Due that, in order to get comparable results, we tested all products with highest settings, if not explicitly advised otherwise by the vendors (as we will use same settings over all tests, the reason is usually that their highest settings either cause too many false alarms, have a too high impact on system performance, or the settings are planned to be changed/removed by the vendor in near future). Here are some notes about the used settings (scan of all files etc. is always enabled) of some products:

• AVIRA, Kaspersky: asked to get tested with heuristic set to high/advanced. Due to that, we recom-mend users to consider also setting the heuristics to high/advanced.
• F-Secure, Sophos: asked to get tested and awarded based on their default settings (i.e. without using their advanced heuristics / suspicious detections setting).
• AVIRA: asked to not enable the informational warnings of suspicious packers. Due that, we did not count them as detections (neither on the malware set, nor on the clean set).

Copyright and Disclaimer

This publication is Copyright © 2011 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.

For more information about AV-Comparatives and the testing methodologies, please visit our website.

AV-Comparatives
(May 2011)