Real-World Protection Test August-November 2012
|Test Period||August - November 2012|
|Number of Testcases||2006|
|Online with cloud connectivity|
|False Alarm Test included|
Malicious software poses an ever-increasing threat, not only due to the number of malware programs increasing, but also due to the nature of the threats. Infection vectors are changing from simple file-based methods to distribution via the Internet. Malware is increasingly focussing on users, e.g. by deceiving them into visiting infected web pages, installing rogue/malicious software or opening emails with malicious attachments.
The scope of protection offered by antivirus programs is extended by the inclusion of e.g. URL-blockers, content filtering, reputation systems and user-friendly behaviour-blockers. If these features are perfectly coordinated with the signature-based and heuristic detection, the protection provided against threats increases.
In this test, all protection features of the product can be used to prevent infection – not just signatures or heuristic file scanning. A suite can step in at any stage of the process – accessing the URL, downloading the file, formation of the file on the local hard drive, file access and file execution – to protect the PC. This means that the test achieves the most realistic way of determining how well the security product protects the PC. Because all of a suite’s components can be used to protect the PC, it is possible for a product to score well in the test by having e.g. very good behavioural protection, but a weak URL blocker. However, we would recommend that all parts of a product should be as effective as possible. It should be borne in mind that not all malware enters computer systems via the Internet, and that e.g. a URL blocker is ineffective against malware introduced to a PC via a USB flash drive or over the local area network.
In spite of these new technologies, it remains very important that the signature-based and heuristic detection abilities of antivirus programs continue to be tested. Even with all the protection features available, the growing frequency of zero-day attacks means that some computers will inevitably become infected. As signatures can be updated, they provide the opportunity to recognize and remove malware which was initially missed by the security software. The newer, “non-conventional” protection technologies often offer no means of checking existing data stores for already-infected files, which can be found on the file servers of many companies. Those new security layers should be understood as an addition to good detection rates, not as a replacement.
The Whole-Product Dynamic “Real-World” Protection test is a joint project of AV-Comparatives and the University of Innsbruck’s Faculty of Computer Science and Quality Engineering. It is partially funded by the Republic of Austria.
For this test, we normally use the Internet security suite, as any protection features that prevent the system from being compromised can be used. However, a vendor can choose to enter their basic antivirus product instead, if they prefer. The main versions of the products tested in each monthly test run are shown below:
- GFI Vipre Internet Security
2012 | 2012 | 2013 | 2013
- PC Tools Internet Security
2013 | 2013 | 2013 | 2013
Testing dozens of antivirus products with hundreds of URLs each per day is a great deal of work, which cannot be done manually (as it would involve visiting thousands of websites in parallel), so it is necessary to use some sort of automation.
Every security program to be tested is installed on its own test computer. All computers are connected to the Internet (details below). The system is frozen, with the operating system and security program in-stalled. The entire test is performed on real workstations. We do not use any kind of virtualization. Each workstation has its own internet connection with its own external IP. We have special agreements with several providers (failover clustering and no traffic blocking) to ensure a stable internet connection. The tests are performed using a live internet connection. We took the necessary precautions (with specially configured firewalls, etc.) not to harm other computers (i.e. not to cause outbreaks).
Hardware and Software
For this test, we used identical workstations, a control and command server and network attached stor-age.
|Workstations||Dell||Optiplex 755||Intel Core 2 Duo||4 GB||80 GB|
|Control Server||Dell||Optiplex 755||Intel Core 2 Duo||8 GB||2 x 500 GB|
|Storage||Eurostor||ES8700-Open-E||Dual Xenon||32 GB||140 TB Raid 6|
The tests were performed under Microsoft Windows XP SP3 with updates as of 1st August 2012. Some further installed software includes:
|Adobe||Flash Player ActiveX||16.0||Microsoft||Office Home||2013|
|Adobe||Flash Player Plug-In||16.0||Microsoft||.NET Framework||4.5.2|
|Microsoft||Internet Explorer||11.0||VideoLAN||VLC Media Player||2.1.5|
Initially we planned to test this year with a fully updated/patched system, but we had to switch back using older/vulnerable/unpatched OS and software versions due to lack of enough exploits in-the-field to test against. This should remind users to keep their systems and applications always up-to-date, in order to minimize the risk of getting infected through exploits using unpatched software vulnerabilities.
In 2013 we plan to perform the test under Windows 7 64 Bit SP1 and more up-to-date software.
We use every security suite with its default (out-of-the-box) settings. Our whole-product dynamic protec-tion test aims to simulate real-world conditions as experienced every day by users. If user interactions are required, we choose allow. If the system will be protected anyway, we count it as blocked even if there was first a user interaction. If the system gets compromised, we count it as user-dependent. We consider â€œprotectionâ€ to mean that the system is not compromised. This means that the malware is not running (or is removed/terminated) and there are no significant/malicious system changes. An out-bound-firewall alert about a running malware process, which asks whether to block traffic form the usersâ€™ workstation to the internet is too little, too late and not considered by us to be protection.
Preparation for every testing day
Every morning, any available security software updates are downloaded and installed, and a new base image is made for that day. This ensures that even in the case the security product would not finish a bigger update during the day (products are being updated before each test case) or is not reachable, it would at least use the updates of the morning, as it would happen to the user in the real world.
Testing Cycle for each malicious URL
Before browsing to each new malicious URL/test-case we update the programs/signatures. New major product versions (i.e. the first digit of the build number is different) are installed once at the begin of the month, which is why in each monthly report we only give the product main version number. Our test software starts monitoring the PC, so that any changes made by the malware will be recorded. Further-more, the recognition algorithms check whether the antivirus program detects the malware. After each test case the machine is reverted to its clean state.
Security products should protect the user’s PC. It is not very important at which stage the protection takes place. It could be while browsing to the website (e.g. protection through URL Blocker), while an exploit tries to run, while the file is being downloaded/created or when the malware is executed (either by the exploit or by the user). After the malware is executed (if not blocked before), we wait several minutes for malicious actions and also to give e.g. behaviour-blockers time to react and remedy actions performed by the malware. If the malware is not detected and the system is indeed infected/compromised, the process goes to “System Compromised”. If a user interaction is required and it is up to the user to decide if something is malicious, and in the case of the worst user decision the system gets compromised, we rate this as “user-dependent”. Because of this, the yellow bars in the results graph can be interpreted either as protected or not protected (it’s up to each individual user to decide what he/she would probably do in that situation).
Due to the dynamic nature of the test, i.e. mimicking real-world conditions, and because of the way several different technologies (such as cloud scanners, reputation services, etc.) work, it is a matter of fact that such tests cannot be repeated or replicated in the way that e.g. static detection rate tests can. Anyway, we log as much data as reasonably possible to support our findings and results. Vendors are invited to provide useful log functions in their products that can provide the additional data they want in the event of disputes. After each testing month, manufacturers are given the opportunity to dispute our conclusion about the compromised cases, so that we can recheck if there were maybe some problems in the automation or with our analysis of the results.
In the case of cloud products, we can only consider the results that the products achieved in our lab at the time of testing; sometimes the cloud services provided by the security vendors are down due to faults or maintenance downtime by the vendors, but these cloud-downtimes are often not disclosed to the users by the vendors. This is also a reason why products relying too heavily on cloud services (and not making use of local heuristics, behaviour blockers, etc.) can be risky, as in such cases the security provided by the products can decrease significantly. Cloud signatures/reputation should be implemented in the products to complement the other local/offline protection features, but not replace them completely, as e.g. offline cloud services would mean the PCs are being exposed to higher risks.
We aim to use visible and relevant malicious websites/malware that are currently out there, and present a risk to ordinary users. We usually try to include as many working drive-by exploits as we find –these are usually well covered by practically all major security products, which may be one reason why the scores look relatively high. The rest are URLs that point directly to malware executables; this causes the malware file to be downloaded, thus replicating a scenario in which the user is tricked by social engineering into following links in spam mails or websites, or installing some Trojan or other malicious software.
We use our own crawling system to search continuously for malicious sites and extract malicious URLs (including spammed malicious links). We also search manually for malicious URLs. If our in-house crawler does not find enough valid malicious URLs on one day, we have contracted some external researchers to provide additional malicious URLs (initially for the exclusive use of AV-Comparatives) and look for additional (re)sources.
In this kind of testing, it is very important to use enough test cases. If an insufficient number of samples are used in comparative tests, differences in results may not indicate actual differences in protective capabilities among the tested products. More details can be found here. Our tests use more test cases (samples) per product and month than any similar test performed by other testing labs. Because of the higher statistical significance this achieves, we consider all the products in each results cluster to be equally effective, assuming that they have a false-positives rate below the industry average.
Hierarchical Cluster Analysis
The dendrogram (using average linkage between groups) shows the results of the hierarchical cluster analysis. It indicates at what level of similarity the clusters are joined. The red drafted line defines the level of similarity. Each intersection indicates a group (in this case 4 groups). Products that had aboveaverage FPs (wrongly blocked score) are marked in red (and downgraded according to the ranking system below).
Below you can see an overview of the individual testing months:
August 2012 – 488 test cases
September 2012 – 519 test cases
October 2012 – 523 test cases
November 2012 – 476 test cases
We purposely do not give exact numbers for the individual months in this report, to avoid the minor differences of a few cases being misused to state that one product is better than the other in a given month and with a given test-set size. We provide the total numbers in the overall reports, where the size of the test-set is bigger, and differences that are more significant may be observed.
False Positive (False Alarm) Test Result
The false-alarm test in the Whole-Product Dynamic “Real-World” Protection Test consists of two parts: wrongly blocked domains (while browsing) and wrongly blocked files (while downloading/installing). It is necessary to test both scenarios because testing only one of the two above cases could penalize products that focus mainly on one type of protection method, either URL filtering or onaccess/behaviour/reputation-based file protection.
Wrongly blocked domains (while browsing)
We used around one thousand randomly chosen popular domains. Blocked non-malicious domains/URLs were counted as false positives (FPs). The wrongly blocked domains have been reported to the respective vendors for review and should now no longer be blocked.
By blocking whole domains, the security products not only risk causing a loss of trust in their warnings, but also possibly causing financial damage (besides the damage to website reputation) to the domain owners, including loss of e.g. advertisement revenue. Due to this, we strongly recommend vendors to block whole domains only in the case where the domain’s sole purpose is to carry/deliver malicious code, and otherwise block just to the malicious pages (as long as they are indeed malicious). Products which tend to block URLs based e.g. on reputation may be more prone to this and score also higher in protection tests, as they may block many unpopular/new websites.
Wrongly blocked files (while downloading/installing)
We used around two thousand different applications listed either as top downloads or as new/recommended downloads from various download portals. The applications were downloaded from the original software developers’ websites (instead of the download portal host), saved to disk and installed to see if they are blocked at any stage of this procedure. Additionally, we included a few clean files that were encountered and disputed over the past months of the Real-World Protection Test.
The duty of security products is to protect against malicious sites/files, not to censor or limit the access only to well-known popular applications and websites. If the user deliberately chooses a high security setting, which warns that it may block some legitimate sites or files, then this may be considered acceptable. However, we do not regard it to be acceptable as a default setting, where the user has not been warned. As the test is done at points in time and FPs on very popular software/websites are usually noticed and fixed within a few hours, it would be surprising to encounter FPs with very popular applications. Due to this, FP tests which are done e.g. only with very popular applications, or which use only the top 50 files from whitelisted/monitored download portals would be a waste of time and resources. Users do not care whether they are infected by malware that affects only them, just as they do not care if the FP count affects only them. While it is preferable that FPs do not affect many users, it should be the goal to avoid having any FPs and to protect against any malicious files, no matter how many users are affected or targeted. Prevalence of FPs based on user-base data is of interest for internal QA testing of AV vendors, but for the ordinary user it is important to know how accurately its product distinguishes between clean and malicious files.
The below table shows the numbers of wrongly blocked domains/files:
|Wrongly blocked clean domains/files
(blocked / user-dependent)
|Wrongly blocked score
[lower is better]
|Qihoo||– / 1 (1)||0.5|
|AVG, Tencent||1 / – (1)||1|
|Avast, Kaspersky||1 / 1 (2)||1.5|
|ESET, G DATA, Panda||2 / – (2)||2|
|Bitdefender||4 / – (4)||4|
|AhnLab, AVIRA, eScan, Fortinet, PC Tools||5 / – (5)||5|
|BullGuard||4 / 2 (6)||5|
|average (6)||average (6)|
|Webroot||7 / 2 (9)||8|
|Trend Micro||10 / – (10)||10|
|Sophos||12 / – (12)||12|
|GFI Vipre||15 / – (15)||15|
|F-Secure||17 / 1 (18)||17.5|
|McAfee||20 / – (20)||20|
 Although user dependent cases are extremely annoying (esp. on clean files) for the user, they were counted only as half for the “wrongly blocked rate” (like for the protection rate).
To determine which products have to be downgraded in our award scheme due to the rate of wrongly blocked sites/files, we backed up our decision by using statistical methods and by looking at the average scores. The following products with above-average FPs have been downgraded: F-Secure, GFI Vipre, McAfee, Sophos, Trend Micro and Webroot
Test period: August – November 2012 (2006 Test cases)
The graph below shows the overall protection rate (all samples), including the minimum and maximum protection rates for the individual months.
|Blocked||User dependent||Compromised||Protection Rate|
[Blocked % + (User dependent % / 2)]
Award levels reached in this Real-World Protection Test
The awards are decided and given by the testers based on the observed test results (after consulting statistical models). Microsoft was considered as a baseline and was therefore tested out-of-competition, due to which it is not included in the awards page. The score of Microsoft Windows Defender would be equivalent to STANDART. The following awards are for the results reached in this Whole-Product Dynamic “Real-World” Protection Test:
* these products got lower awards due to false alarms
Most operating systems already include their own firewalls, automatic updates, and may even ask the user before downloading or executing files if they really want to do that, warning that download-ing/executing files can be dangerous. Mail clients and web mails include spam filters too. Furthermore, most browsers include Pop-Up blockers, Phishing/URL-Filters and the ability to remove cookies. Those are just some of the build-in protection features, but despite all of them, systems can get infected anyway. The reason for this in most cases is the ordinary user, who may get tricked by social engineering into visiting malicious websites or installing malicious software. Users expect a security product not to ask them if they really want to execute a file etc. but expect that the security product will protect the sys-tem in any case without them having to think about it, and despite what they do (e.g. executing un-known files). We try to deliver good and easy-to-read test reports for end-users. We are continuously working on improving further our automated systems to deliver a better overview of product capabilities.
Copyright and Disclaimer
This publication is Copyright © 2012 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.
For more information about AV-Comparatives and the testing methodologies, please visit our website.