Android Test 2017 – 100+ Apps

Introduction

In April 2014, the website Android Police (www.androidpolice.com) published the results of their investigation into an Android app called Virus Shield. At the time, the app had been downloaded over 10,000 times, and was the most successful new paid-for app and third most successful paid-for app overall. It had also received an impressive 4.7 out of 5 rating from users. As its name suggests, Virus Shield was supposed to be an antimalware app for Android devices. However, when Android Police investigated the app, they found that it had no antivirus functions at all, and that tapping on the icon supposed to activate the protection does literally nothing except change the icon from a cross to a tick (checkmark). The only true claims made by the developer were that it had minimal effect on battery life and did not display advertisements.

On discovering that the app was a scam, Google removed Virus Shield from the Play Store, suspended the developer’s account, and refunded users who had purchased the app. This means that on this occasion, little or no harm was done, but it shows clearly how easy it is to produce a poorly performing app and make it successful by means of good marketing. Android Police should be congratulated for discovering this scam; they point out that it would be difficult for Google to stop all such scams, and that more rigorous testing of apps available on the Play Store would make the store less open than it is now. It should be noted that it is easier to spot a malicious app – due to suspicious code – than a useless app like Virus Shield, which is not in itself harmful. There is also the possibility that thorough scrutiny of apps before they can be released might be prohibitively expensive and/or time-consuming. Google’s advice to check the ratings of an app before purchasing it is in principle good, but clearly it would not have helped in this case – regardless of whether the overwhelmingly positive reviews were fakes posted by the developer, or genuine reviews posted by duped users. Of the apps tested for this report, practically all had a rating of 4 or higher, even though a number of them turned out to be ineffective.

In the case of antimalware apps, there is a straightforward solution: testing the apps against real malware samples by independent research labs. The aim of this test is to find out which of the antimalware apps for Android in the Google Play Store are genuine and effective, and to expose any that are ineffective or just fake. This report was commissioned by xxx magazine.

Tested Products

For this test, we searched for and downloaded over 100 antimalware security apps of different developers from the Google Play Store.

The following 110 security apps were analyzed:

ADV Antivirus Mobile Agency	Kaspersky Antivirus & Security
AegisLab Antivirus Premium	LINE Antivirus
AhnLab V3 Mobile Security	LionMobi Power Security Antivirus Clean
AiDevLab Security Antivirus Max Clean	Live multi Player Game Antivirus Total Security
AndroHelm AntiVirus	Lookout Antivirus & Security
Antiy AVL	MalwareBytes Anti-Malware
Ascal Antivirus & Mobile Security	Max Security Antivirus PRO
Avast Mobile Security & Antivirus	McAfee Security & Antivirus
AVC Security Antivirus Clean	MediaCenterSocial Antivirus
AVG Antivirus PRO	Melodiu Ideas LuLa Antivirus Malware Protect
AVIRA Antivirus Security	NCN-NetConsulting Free Antivirus Clean&Boost
Baboon Antivirus	Netlink Mobile Antivirus Pro
Baidu DU Antivirus Mobile Security & AppLock	NguyenManh Antivirus Security
Bastiv Security Antivirus	NOAH Security Antivirus
Bitdefender Mobile Security & Antivirus	NQ Mobile Security & Antivirus
BitInception Antivirus	NSHC Ariasecure Bornaria security (Antivirus)
BKAV Security Antivirus Free	One App Super Clean Speed Security MAX
Bluesteeleffect Studios Antivirus Security Cleaner Pro	Panda Free Antivirus
Brainiacs Apps Antivirus System	Perfect Tools Antivirus
BuildOut Tech Antivirus	Play Studio Apps Mobile Security Antivirus
BullGuard Mobile Security and Antivirus	Playnos Yalp Security Antivirus
CA Uber Apps Security Antivirus Android	Pocao Antivirus
Check Point ZoneAlarm Mobile Security	Poke And Touch Security Antivirus
Cheetah Mobile CM Security CleanMaster	Pro Tool Apps Antivirus Security
CHOMAR Antivirus Security	Psafe Antivirus
Comodo Mobile Security	Qihoo 360 Mobile Security
Cora Mobile Antivirus	Quick Heal Antivirus & Mobile Security
CTPlate Free Antivirus	Quicken Security Studio Smart Antivirus
CY Security Antivirus Cleaner	REVE Antivirus Mobile Security
Defenx Security Suite	Security Defend Total Antivirus Defender PRO
DevByMe MDD Guard Antivirus & Antispyware	Smartdev Studio Security Antivirus
Dr.Web Antivirus Light	Sophos Free Antivirus and Security
Duc Nguyen FJC Antivirus Spy Mobile Security Pro	SPAMfighter VIRUSfighter Antivirus
Emsisoft Mobile Security	Stock VIP Antivirus
EnjoyPlus Security Antivirus	Super Security Tech Ace Security Plus Antivirus
eScan Mobile Security	SuperSoftDev Antivirus
ESET Mobile Security & Antivirus	Symantec Norton Antivirus & Security
EveryZone Turbo Vaccine Mobile	Taolee Antivirus
Farga Security Antivirus	Tencent WeSecure Antivirus
Fast Track Super Security Free AntiVirus	TG Soft VirIT Mobile Security
Fotoable Photo Editor Creative Cleaner&Security&Applock	TiTanTech CleaningVirus 360
F-Secure Mobile Security	Total Defense Mobile Security
G DATA Internet Security	Trend Micro Mobile Security & Antivirus
GO Security Antivirus Applock	TrustGo Antivirus & Mobile Security
Gpaddy Antivirus Pro	Trustlook Premium Mobile Antivirus
Green Booster Antivirus	Vasa Virus Seeker Mobile Security
Guaraw Yadaw Antvirus Security Shield	Viettel Antivirus Free Mobile Security
H2 Free Antivirus	VSAR Total Virus Scanner & Remover
Hi Dev Team Security Antivirus & Privacy	Webroot Security Premier
Hornet Antivirus PRO	WeMakeItAppen Antivirus Fast
Ikarus mobile.security	WhiteArmor Security Pro
IncodeSolutions Anti-Malware	Z Security Apps Studio Virus Cleaner Antivirus
Iobit AMC Security	Zemana Mobile Antivirus
Itus Antivirus	Zillya! Internet Security & Antivirus
K7 Mobile Security	ZONER Mobile Security

The antimalware apps from the following 9 vendors were so buggy that they could not be installed/tested: CY Security, DevByMe, Gauraw Yadaw, Live multi Player Game, MediaCenterSocial, NguyenManh, REVE, SPAMfighter, and SuperSoftDev.

The antimalware apps from the following 5 vendors pose risks, as they contain unsafe features, collect sensitive data, or deceive users by claiming to use specific well-known and effective antivirus engines, although in fact they do not: Cora, Hi Dev Team, Melodiu Ideast, Netlink and Z Security Apps Studio.

The antimalware apps of the following 10 vendors have in the meantime already been removed by Google from the Play Store: BuildOut Tech, Duc Nguyen, EveryZone, Perfect Tools, Playnos Yalp, Poke And Touch, Quicken, Stock, Taolee and TiTanTech.

Most of the apps removed by Google, as well as the heavily buggy or unsafe apps/apps with low protection scores, appear to have been developed either by amateur programmers, or by software manufacturers that are not focused on the security business (i.e. develop all kinds of apps, and/or are in the advertisement/monetization business). Apps made by amateurs can be often spotted in the Google Play Store by looking at the options for contacting the authors. Typically, hobby developers will not provide a website address, merely an email address (usually Gmail, Yahoo, etc.).

Additionally, most such apps do not provide any sort of privacy policy. Google is planning to purge from the Play Store all apps which lack a privacy policy, which could help to get rid of some low-quality apps. Of course, one should bear in mind that not all apps made by amateur developers are necessarily ineffective.

Test Procedure

Description of test system

The Android security solutions tested were checked for their efficacy in protecting against the top 1,000 most common Android malware threats of 2016. Manually testing 100+ security products against 1,000 malicious apps is not practicable. Because of that, the test was run on our automated Android testing framework.

Even though the testing process is automated, the framework realistically simulates real-world conditions. This includes testing on physical Android devices (as opposed to emulators), as well as simulation of realistic device usage patterns.

The framework consists of two components: a client app on each of the test devices, and a server application. The client app monitors the status of the device and sends its findings to the server at the end of a test case to document the testing process. The client monitors file and process changes,  newly installed apps and their permissions, as well as reactions of the installed security software to malicious activities on the device. The server remotely controls the test devices via WiFi and organizes the results received by the client applications.

During the test, each security app was installed on a separate test device.

The system scales well with the number of connected clients. This allows a large number of security products to be tested in parallel. To ensure even chances for all participating products, connected clients can be synchronized to start the execution of a test case at the same time. This is especially important for testing recent malware samples, which security vendors may not have encountered yet.

Methodology

The test was performed on the 12th of January 2017, on Nexus 5 devices running Android 6.0.1 (“Marshmallow”). Each security app was installed on a separate physical test device. Before the test was started, the software testbed on all test devices – Android itself, stock Android apps, plus testing-specific third-party apps – was updated. After this, automatic updates were switched off, thus freezing the state of the test system. Next, the security apps to be tested were installed and started on their respective devices, updated to the latest version where applicable, and the malware definitions brought fully up to date.

If any security application encouraged the user to perform certain actions to secure the device, such as running an initial scan, these actions were performed. If the application contained protection functions such as on-install scanning, cloud protection, or detection of Potentially Unwanted Applications (PUA), these features were activated as well. To ensure that all security products could access to their respective cloud analysis services, each device was connected to the internet via a WiFi connection.

Once these steps were taken, a clean snapshot of each device’s storage was created, and the test was started.

Each test case was conducted using the same process:

1. Open the Chrome browser and download the malicious sample
2. Open the downloaded .apk file using a file explorer app
3. Install the malicious app
4. Execute the installed app

After each of the above steps, the installed security application was granted enough time to analyze the malicious sample and notify the user of malicious activity on the device.

If, at any point during the execution of a test case, the installed Anti-Virus application detected and blocked the malicious sample, the sample was considered “detected” and the test case was concluded (apps detected after installation were not executed, for instance).

At the end of each test case, the device was reset to a clean state. If the malicious sample was not executed on the device, the sample was uninstalled and/or deleted from the device storage. If the malicious sample was run, the clean device snapshot was restored before starting the next test case.

When calculating the protection score for each product, we did not distinguish between different detection times during a test case (e.g., after download vs. after install). The only aspect influencing the protection rate is whether the security solution protected the device from being compromised by the malicious sample.

A basic false-alarm test was done, just to check that none of the antimalware products “protects” the system by simply identifying all new apps as malicious. None of the apps tested detected any of 50 popular installed on a clean Android device as malware.

Testcases

For this test, the Top 1,000 most common Android malware threats of 2016 were used. With such samples, detection rates of between 90% and 100% should be easily achieved by genuine and effective antimalware apps.

Number of tested apps	110
Number of tested malicious APKs	1000
Number of tested clean APKs	50

In total, around 100,000 test runs have been performed for this report.

Test Results

Vendor	%
AhnLab	100,0%
Antiy
Avast
AVG
AVIRA
Baidu DU Apps
Bitdefender
BullGuard
Cheetah Mobile
Emsisoft
ESET
G DATA
Ikarus
Kaspersky Lab
McAfee
One App
Psafe
Quick Heal
Sophos
Symantec
Tencent
Total Defense
Trend Micro
WhiteArmor
BKAV	99,9%
Webroot
Bastiv	99,8%
Qihoo 360
TrustGo	99,6%
Dr.Web	99,5%
F-Secure	98,9%
ADV	98,2%
Green Booster
Pocao
Avc Security	97,5%
NOAH Security
Fast Track	97,4%
Fotoable	96,6%
Iobit
MalwareBytes	96,0%
eScan	95,8%
AiDevLab	95,7%
NSHC	95,5%
K7	95,2%
Panda	95,1%
ZoneAlarm	94,6%
Lookout	92,2%
AegisLab	90,2%

 

Zemana	88,8%
TG Soft	88,6%
ZONER	83,9%
Comodo	70,6%
Gpaddy	63,0%
GO Security	61,4%
Trustlook
Viettel	52,9%
Super Security Tech	48,6%
NQ	48,1%
LionMobi	46,6%
LINE	39,9%
Zillya!	34,4%
CA Uber Apps	33,9%
WeMakeItAppen

 

AndroHelm	0 – 30%
Ascal
Baboon
BitInception
Bluesteeleffect Studios
Brainiacs Apps
CHOMAR
CTPlate
Defenx
EnjoyPlus
Farga
H2
Hornet
IncodeSolutions
Itus
Max Security
NCN-NetConsulting
Play Studio Apps
Pro Tool Apps
Security Defend
SmartDev Studio
Vasa
VSAR

The table above shows the protection rates reached by the products of  the respective vendors. We consider apps scoring below 30% on common Android threats to be unsafe and completely unacceptable.

The anti-malware apps of AndroHelm, Ascal, Baboon, BitInception, Bluesteeleffect Studios, Brainiacs Apps, CHOMAR, CTPlate, Defenx, EnjoyPlus, Farga, H2, Hornet, IncodeSolutions, Itus, Max Security, NCN-NetConsulting, Play Studio Apps, Pro Tool Apps, Security Defend, SmartDev Studio, Vasa and VSAR detected between 0% and 30% of the 1,000 malicious Android apps, and are not listed in the chart above – partly for display reasons, but also because they are ineffective.

Notes

Some products make use of other engines (see examples below). While some score the same, some of them score differently despite making use of the same engine. According to the licensing developers, this is often the case due to several factors, such as other internal settings used by the third-party apps, the use of older engines or additional engines, engine implementation and bugs.

• AiDevLab makes use of the Tencent
• GO Security makes use of the Trustlook engine.
• PSafe is using an engine from Qihoo.
• Iobit, Fotoable, One App, WeMakeItAppen and CA Uber Apps make use of the OpenAVL
• TotalDefense uses and appears to be a rebranded version of Bitdefender.
  
  

• ADV and Pocoa are basically the same (but claim to be from different developers):
  
  

• Bluesteeleffect Studios, EnjoyPlus, Brainiacs Apps, ProTool Apps are basically all the same (claim to be from different developers). All of them detected 0% of the used malware test-set.
  

Some apps claim to use reputable engines, but in fact do not; these must be regarded as scams. An example of an advertising text used by an app which was amongst the 5 apps weeded out of the test for blatant dishonesty, is shown in the screenshot below.

We performed sufficient testing with this app to be certain that it does not use the McAfee engine. McAfee have also informed us that they are not related in any way to those developers, there is no partnership of any kind, and that they have not licensed the developer to make use of their engine. In other words, it is a scam. There are some other products using very similar advertising text, which must also be considered scams. Scam apps can be reported to Google; unfortunately, even if Google removes such apps from the store, they will usually re-appear very soon under different names.

Conclusion

Amongst the security apps available from the Google Play Store there are a few which have so many bugs that they either cannot be installed, or crash so frequently as to be unusable.

Some  of the Android security products detected far too few of the malware samples in our test – in some cases literally nothing – to be recommended as anti-malware apps. In a few cases, this might be due to apps having been abandoned by the developer and thus no longer being updated in the Play Store. Whilst such cases cannot be regarded as scams, we consider it irresponsible of the developers not to remove these apps from the Store.

A few products from relatively well-known vendors did not score very well. It is possible that the manufacturers have developed them purely for marketing reasons. That is to say, there is not much money in the Android security-app market, but having an Android app visible in the Google Play Store helps to keep the vendor visible, and may thus promote their other, more profitable products such as Windows security programs.

24 of the products we tested detected 100% of the malware samples; considering that the most common malicious Android apps of 2016 were used, this is what they should do. Most of the vendors that usually take part in independent tests score highly, as their products are regularly scrutinised, and they actively develop them to ensure they are effective.

When it comes to choosing an Android security app, we recommend considering the following factors. Using user ratings is clearly not effective, as the vast majority of users will give their rating based solely on the user experience, without having any idea as to whether the app offers effective protection. Some other reviews will have been faked by developers. Practically all the 110 apps we looked at had a review score of 4 or higher on the Google Play Store. Similarly, the number of downloads can only be a very rough guide; a successful scam app may be downloaded many times before it is found to be a fake. Using well-known and reputable, verified vendors is recommended. As well as participating in tests by independent test institutes, such vendors will have a professional website with contact information and a privacy policy. It should also be possible to try the app – typically some few weeks trial use is allowed – before purchasing. Users can then assess the usability and any additional features of the product. A number of vendors make very effective free versions of their apps; generally these are more likely to display advertising than the paid version, though this is not always the case.

For additional Android security app tests and reviews, please see:
https://www.av-comparatives.org/testmethod/mobile-security-reviews/

Copyright and Disclaimer

This publication is Copyright © 2017 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.

For more information about AV-Comparatives and the testing methodologies, please visit our website.

AV-Comparatives
(February 2017)