Mobile Security Test March 2015
| Release date | 2015-03-31 |
| Revision date | 2015-03-30 |
| Test Period | February 2015 |
| Number of Testcases | 4523 |
| Online with cloud connectivity |  |
| Update allowed | |
| False Alarm Test included | |
| Platform/OS | Android Mobile |
Introduction
This test covers security products for smartphones and tablets running Google’s Android operating system. The report covers details of the products made by leading manufacturers who have agreed to participate. The test was conducted in February 2015 on identical LG Nexus 5 smartphones running Android 4.4.4.
Please note that we will be conducting our main test and full review of mobile security products this summer, as usual. This report, which covers malware protection only, allows vendors to test new products before deciding whether to join the main public test and review later in the year. Participating vendors were allowed to decide whether to have their results published, but had to do this before the test was carried out. Thus the vendors shown in the results list were all confident of their respective products’ abilities; several other vendors took part, but with the condition that the results would be kept internal.
The products that participated in this year’s test are listed below. The manufacturers either provided us with the latest version of their product available on their website or several third party stores, or confirmed that it was available from the Google Play Store at the time of the test (February 2015). Most of the tested products can either be found on Google play or in case of Anguanjia on their website and on several big Chinese third-party stores. The tested Qihoo 360 version is only available on the Qihoo 360 website.
Tested Products
Anguanjia 安全管家 5.0.0
Test Procedure
Google Android contains basic protection against malware out-of-the-box. If the user installs new software on his phone, the phone asks the Google Safebrowsing API to check whether the app is malicious or not. Our intention was to identify the detection rate of this service for all malicious samples in the test. After some tests, we found out that the results vary, even if we always perform the test with the same malicious sample. After some research, we found that the number of requests to the API of Google Safebrowsing is limited. Google does not provide absolute values for the number of requests. The API documentation states “In order to ensure high availability of the API, Google limits the frequency of client requests. This is handled differently depending on the type of request”.
To double-check our findings we set up a man-in-the-middle attack and analysed the network traffic while installing new applications. We could confirm the findings from previous tests with this method, as the responses from the Google Safebrowsing API were different, even if we always installed the same application. In the case of a detection, the API showed a description of the string on the display, such as “It has been modified to include potentially harmful code” and additional data. In the case of a miss the response was just a two byte long response (“08 00” in Hex).
As the size of the sample set was big, we could not get any coherent detection rates for the Google Safebrowsing service. Requests to the security team of Google to whitelist our devices for those kind of limitations were not answered in time.
Testcases
We collected the malware samples used in the test during the period of few months leading up to the test. 4,523 malicious applications were used to create a representative test set. So-called “potentially unwanted apps” were removed from the test-set. The test-set consisted of 125 main malware families.
The security products were updated and tested on the 23rd February 2015. The test was conducted with an active Internet connection on genuine Android smartphones (no emulators were used). The test set consisted exclusively of .APK files. An on-demand scan was conducted first. After this, every undetected app was installed manually. We did this to allow the products to detect the malware using real-time protection.
Test Results
The chart below shows, that the protection rates against real Android malware are very high. This might be due to the increasingly aggressive detection by app reputation for apps that are not on Google Play, but maybe also because many of the participants in our test are leading mobile security vendors with good protection rates.
We additionally conducted a false-positive test using the top 200 ad-free programs from various popular app stores. Only Avast produced a single false alarm when installing those apps, all other programs in this test had none. We did however notice that all products are prone to false alarms and misidentification especially of Chinese apps and/or apps obtained outside of well-known stores. In Asia, perceptions of what constitutes unwanted apps may vary from those in Europe or North America. We removed disputed questionable or controversial samples from the test-sets.
Notes
This test only covers malware protection. Users looking for a comprehensive review of mobile security products, including features, functionality and user interface, can consult our September 2014 review , or wait until our next report is published (September of this year).
The perfect mobile-security product does not yet exist. As with Windows products, we recommend drawing up a short list after reading about the advantages and disadvantages of each product in our review. A free trial version of each candidate product can then be installed and tested for a few days; this should make the decision easier. For Android security products in particular, new versions with improvements and additional functionality are constantly being released.
The antimalware component of a mobile security product scans the mobile device for malicious software, which it deletes or quarantines. For this function to work effectively it has to be kept up-to-date; some products also make use of the cloud when scanning. When travelling abroad, users need to be careful that automatic updates and cloud scans do not incur high roaming costs from the mobile service provider.
Copyright and Disclaimer
This publication is Copyright © 2015 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.
For more information about AV-Comparatives and the testing methodologies, please visit our website.
AV-Comparatives
(March 2015)