Retrospective / Proactive Test May 2010
Heuristic and behavioural protection against new/unknown malicious software
|Test Period||February 2010|
|Number of Testcases||27271|
|Online with cloud connectivity|
|False Alarm Test included|
- Kingsoft Antivirus 2010
- PC Tools Spyware Doctor with AV 7.0
Anti-Virus products often claim to have high proactive detection capabilities – far higher than those reached in this test. This is not just a self-promotional statement; it is possible that products reach the stated percentages, but this depends on the duration of the test-period, the size of the sample set and the used samples. The data shows how good the proactive detection capabilities of the scanners were in detecting new threats. Users should not be afraid if products have, in a retrospective test, low percentages. If the anti-virus software is always kept up-to-date, it will be able to detect more samples. For understanding how the detection rates of the Anti-Virus products look with updated signatures and programs, have a look at our regular on-demand detection tests. Only the on-demand detection capability was tested. Some products may be had the ability to detect some samples e.g. on-execution or by other monitoring tools, like behaviour-blocker, etc. Those kinds of additional protection technologies are considered by AV-Comparatives in e.g. dynamic tests.
This test report is the second part of the February 2010 test. The report is delivered begin of June due the high-required work, deeper analysis and preparation of the retrospective test-set. Many new viruses and other types of malware appear every day, this is why it’s important that Anti-Virus products not only provide new updates, as often and as fast as possible, but also that they are able to detect such threats in advance (also without executing them) with generic and/or heuristic techniques. Even if nowadays most Anti-Virus products provide daily, hourly or cloud updates, without heuristic/generic methods there is always a time-frame where the user is not reliably protected.
The products used the same updates and signatures they had the 10th February, and the same highest detection settings were used as in February. This test shows the proactive detection capabilities that the products had at that time. We used new malware appeared between the 11th and 18th February 2010. The following 20 products were tested:
We tried to include in the test-set only prevalent real-world malware that has not been seen before the 10th February 2010 by consulting telemetry / cloud data collected and shared within the AV industry. Consulting that data was quite interesting for us, as it showed that, while some vendors had seen some malware already many months or even years ago, the same malware hashes appeared in some other vendors clouds only recently.
The awards are given by the testers after consulting a number of statistical methods, including hierarchical clustering. We based our decisions on the following scheme:
|None - Few FPs|
|Very many FPs|
|Crazy many FPs|
The results show the proactive (generic/heuristic) on-demand detection capabilities of the scan engines against new malware. This Test is performed on-demand, it is NOT an on-execution/behavioral test. The percentages are rounded to the nearest whole number. Do not take the results as an absolute assessment of quality – they just give an idea of who detected more, and who less, in this specific test. To know how these anti-virus products perform with updated signatures, please have a look at our on-demand tests of February and August. Readers should look at the results and build an opinion based on their needs. All the tested products are already selected from a group of very good scanners and if used correctly and kept up-to-date, users can feel safe with any of them.
False Positive (False Alarm) Test Result
To better evaluate the quality of the detection capabilities, the false alarm rate has to be taken into account too. A false alarm (or false positive) is when an Anti-Virus product flags an innocent file to be infected when it is not. False alarms can sometimes cause as much troubles like a real infection.
The false-alarm test results were already included in the February test report. For details, please read the report, False Alarm Test February 2010.
|1.||eScan||1||very few FPs (0-3)|
|3.||Bitdefender, ESET, Microsoft||3|
|4.||Sophos||4||few FPs (4-15)|
|5.||G DATA, Kaspersky Lab||5|
|9.||Avast, Avira, Symantec||11|
|10.||Trend Micro||38||many FPs (over 15)|
The results show the proactive protection capabilities of the various products against new malware. The percentages are rounded to the nearest whole number.
Below you can see the proactive protection results over our set of new and prevalent malware files/families appeared in-the-field (27,271 malware samples):
|Blocked||Compromised||Proactive / Protection Rate||False Alarms||Cluster|
|ESET NOD32, F-Secure||14181||13090||52%||very few||1|
|BitDefender, eScan||13636||13636||50%||very few||1|
Award levels reached in this Heuristic / Behavioural Test
The following awards are for the results reached in the proactive/behavioural test, considering not only the protection rates against new malware, but also the false alarm rates:
* these products got lower awards due to false alarms
To avoid some frequent questions, below are some notes about the used settings (scan of all files etc. is always enabled) of some products, whereas highest settings were not used on vendors request:
- F-Secure, Sophos: asked to get tested and awarded based on their default settings (i.e. without using their advanced heuristics / suspicious detections setting).
- AVG, AVIRA: asked to do not enable/consider the informational warnings of packers as detections.
Copyright and Disclaimer
This publication is Copyright © 2010 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.
For more information about AV-Comparatives and the testing methodologies, please visit our website.