False Alarm Test March 2016

Introduction

This report is an appendix to the File Detection Test September 2016 listing details about the discovered False Alarms.

With AV testing it is important to measure not only detection capabilities but also reliability. One aspect of reliability is the ability to recognize clean files as such, and not produce false alarms (false positives). No product is immune from false positives (FPs), but some produce more than others, and the our goal is to find out which programs do best in this respect. There is no complete collection of all legitimate files that exist, and so no “ultimate” test of FPs can be done. What can be done, and is reasonable, is to create and use a set of clean files which is independently collected. If with such a set one product has e.g. 30 FPs and another only 5, it is likely that the first product is more prone to FP’s than the other. It doesn’t mean the product with 5 FPs doesn’t have more than 5 FPs globally, but it is the relative number that is important.

Tested Products

• Avast Free Antivirus 11.1Build: 11.1.2245
• AVG Internet Security 2016Build: 2016.0.7442
• Avira Antivirus Pro 15.0Build: 15.0.16.282
• Bitdefender Internet Security 20.0Build: 20.0.25.1377
• BullGuard Internet Security 16.0Build: 16.0.315.1
• Emsisoft Anti-Malware 11.0Build: 11.0.0.6191
• eScan Internet Security 14.0Build: 14.0.1400.1831
• ESET Smart Security 9.0Build: 9.0.349
• F-Secure SAFE 14.150Build: 14.150.101
• Fortinet FortiClient 5.2Build: 5.2.5.0658
• Kaspersky Internet Security 16.0Build: 16.0.1.445 (a)
• Lavasoft Ad-Aware Pro 11.10Build: 11.10.767.8917
• McAfee Internet Security 18.0Build: 18.0.5011
• Microsoft Defender 4.9Build: 4.9.10586.0
• Quick Heal Total Security 16.0Build: 16.00.9.0.24.4
• Sophos Endpoint Security and Control 10.3Build: 10.3.15.69
• Tencent PC Manager 11.2Build: 11.2.26147.901
• ThreatTrack Vipre Internet Security 9.3Build: 9.3.2.17
• Trend Micro Internet Security 10.0Build: 10.0.1186

Test Procedure

In order to give more information to the users about the false alarms, we try to rate the prevalence of the false alarms. Files which were digitally signed are considered more important. Due to that, a file with e.g. prevalence “level 1” and a valid digital signature is upgraded to the next level (e.g. prevalence “level 2”). Files which according to several telemetry sources had zero prevalence have been provided to the vendors in order to fix them, but have also been removed from the set and were not counted as false alarms.

The prevalence is given in five categories and labeled with the following colors: 

Level		Presumed number of affected users	Comments
1		Probably fewer than hundred users	Individual cases, old or rarely used files, very low prevalence
2		Probably several hundreds of users	Initial distribution of such files was probably much higher, but current usage on actual systems is lower (despite its presence), that is why also well-known software may now affect / have only a prevalence of some hundreds or thousands of users.
3		Probably several thousands of users
4		Probably several tens of thousands (or more) of users
5		Probably several hundreds of thousands or millions of users	Such cases are likely to be seen much less frequently in a false alarm test done at a specific time, as such files are usually either whitelisted or would be noticed and fixed very fast.

Most false alarms will probably fall into the first two levels most of the time. In our opinion, anti-virus products should not have false alarms on any sort of clean files regardless of how many users are currently affected by them. While some AV vendors may play down the risk of false alarms and play up the risk of malware, we are not going to rate products based on what the supposed prevalence of false alarms is. We already allow a certain amount of false alarms (currently 10) inside our clean set before we start penalizing scores, and in our opinion products which produce a higher amount of false alarms are also more likely to produce false alarms on more prevalent files (or in other sets of clean files). The prevalence data we give about clean files is just for informational purpose. The listed prevalence can differ inside the report, depending on which file/version the false alarm occurred, and/or how many files of the same kind were affected.

Testcases

All listed false alarms were encountered at the time of testing. False alarms caused by unencrypted data blocks in anti-virus related files were not counted. If a product had several false alarms belonging to the same software, it is counted here as only one false alarm. Cracks, keygens, etc. or other highly questionable tools, including FPs distributed/shared primarily by vendors (which may be in the several thousands) or other non-independent sources are not counted here as false positives.

Test Results

Some products using third-party engines/signatures may have fewer or more false alarms than the licensed engine has by its own, e.g. due to different internal settings implemented, the additional checks/engines/clouds/signatures, whitelist databases, time delay between the release of the original signatures and the availability of the signatures for third-party products, additional quality assurance of signatures before release, etc.

False Positives (FPs) are an important measurement for AV quality.  One FP report from a customer can result in large amount of engineering and support work to resolve the issue.  Sometimes this can even lead to important data loss or system unavailability.  Even “not significant” FPs (or FPs on old applications) deserve mention and attention because FPs are likely to be a result of principled rule detections.  It just happened that the FP was on an insignificant file. The FP possibility is probably still in the product and could cause an FP again on a more significant file. Thus, they still deserve mention and still deserve to be penalised. Below you will find the false alarms we observed in our independent set of clean files. Red entries highlight false alarms on files that were digitally signed.

1.	ESET, Trend Micro	0	very few FPs
2.	McAfee	1	few FPs
3.	BullGuard, eScan, Sophos	2
4.	Bitdefender, Emsisoft, Kaspersky, ThreatTrack	3
5.	F-Secure, Lavasoft, Tencent	4
6.	Quick Heal	8
7.	Avira	9	many FPs
8.	AVG	10
9.	Fortinet, Microsoft	13
10.	Avast	17

Details about the discovered false alarms

ESET and Trend Micro had zero false alarms on the used set of clean files.

McAfee		1 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
TVgenial package	Artemis!E1DB26418B72

 

BullGuard		2 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Granny package	Gen:Variant.Razy.19282
Runner package	Gen:Variant.Barys.49628

 

eScan		2 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Corel package	Gen:Variant.Barys.52348 (DB)
Runner package	Gen:Variant.Barys.49628 (DB)

 

Sophos		2 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
BZIP package	Mal/Dorf-D
TNI package	Mal/Generic-L

 

Bitdefender		3 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Corel package	Gen:Variant.Barys.52348
Granny package	Gen:Variant.Razy.19282
Runner package	Gen:Variant.Barys.49628

 

Emsisoft		3 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Corel package	Gen:Variant.Barys.52348 (B)
Granny package	Gen:Variant.Razy.19282 (B)
Runner package	Gen:Variant.Barys.49628 (B)

 

Kaspersky Lab		3 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
OnlineEye package	Trojan-Downloader.Win32.Banload.aajbo
Puzzle package	Trojan-Spy.Win32.Taopap.phe
Radeon package	P2P-Worm.Win32.Palevo.hynv

 

ThreatTrack		3 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Corel package	Gen:Variant.Barys.52348
Granny package	Gen:Variant.Razy.19282
Runner package	Gen:Variant.Barys.49628

 

F-Secure		4 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Corel package	Gen:Variant.Barys.52348
FinePrint package	Trojan:W32/Gen4135.1fc23018e8!Online
Runner package	Gen:Variant.Barys.49628
Xtreme package	Trojan-dropper:W32/Coinminer.99db20ce3c!Online

 

Lavasoft		4 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Corel package	Gen:Variant.Barys.52348
Granny package	Gen:Variant.Razy.19282
Mame package	Gen:Variant.Barys.52421
Runner package	Gen:Variant.Barys.49628

 

Tencent		4 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Corel package	Gen:Variant.Barys.52348
Granny package	Gen:Variant.Razy.19282
Mame package	Gen:Variant.Barys.52421
Runner package	Gen:Variant.Barys.49628

 

Quick Heal		8 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Elsword package	Trojanspy.Agent.018127
Granny package	EE:Malwr.Heur.Razy.19282
IronBrowser package	JS/Agent.KK
MakeDisk package	Ransom.Crowti.A4
PerfectMenu package	Trojan.Malagent.019169
Runner package	EE:Malwr.Heur.Barys.49628
Screensaver package	Trojan.Scar.013919
WB package	Suspicious

 

AVIRA		9 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
AudaPad package	HEUR/APC
Chilkat package	HEUR/APC
CreateAMall package	HEUR/APC
CueMaker package	HEUR/APC
Drei package	HEUR/APC
Fujitsu package	HEUR/APC
PlantsVSZombies package	HEUR/APC
Tiscali package	HEUR/APC
WinHotel package	HEUR/APC

 

AVG		10 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Acer package	Zbot.AJKR
AirSnare package	Collected_c.CGRB
ArrowSearch package	Win32/DH{d4IRgQw}
DigitalTheatre package	Win32/DH{cjETMHmBRg?}
MightyChicken package	Win32/DH{gVGBCoFT?}
PowerTranslator package	Win32/DH{ZzWCHIEPgRxB?}
SIW package	Generic36.CGMO
SysTrayX package	Agent5.AKKG
VirtualExpander package	Win32/DH{gg92A1g?}
Zattoo package	Win32/Herz

 

Fortinet		13 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
ASUS package	W32/Agent.NESVWS!tr
CableMon package	W32/Generic.AC.2181457
ColdFusion package	W32/Generic.AC.2506367
HWinfo package	W32/Bayrob.AT!tr
Macromedia package	W32/Generic.AC.2506367
PageDfrg package	PossibleThreat.SB!tr.rkit
Pi package	W32/Kryptik.EKOM!tr
SkinPack package	W32/Sim.SP!tr
Startupo package	W32/Generic.AC.256673
SysOpt package	INF/Qhost!tr
Triton package	W32/Generic.AC.2926293
WireShark package	W32/Kryptik.EMEK!tr
WS_FTP package	W32/Kryptik.ELYI!tr

 

Microsoft		13 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
2H4U package	Trojan:Win32/Varpes.J!plock
Battlefield package	Trojan:Win32/Skeeyah.A!bit
ClipInc package	Trojan:Win32/Dorv.C!rfn
Dbox package	Trojan:Win32/Varpes.J!plock
DerLauncher package	Trojan:Win32/Varpes.K!plock
Fotokasten package	Trojan:Win32/Varpes.J!plock
HiddenFinder package	Trojan:Win32/Varpes.K!plock
KeriverImage package	Trojan:Win32/Varpes.J!plock
MediaCenter package	Trojan:Win32/Varpes.J!plock
MoviePlus package	Trojan:Win32/Varpes.K!plock
Nero package	Trojan:Win32/Varpes.J!plock
OrgaMax package	Trojan:Win32/Varpes.J!plock
Outlookers package	Trojan:Win32/Varpes.J!plock

 

Avast		17 False Alarms
False alarm found in some parts of	Detected as	Supposed prevalence
Adobe package	Win32:GenMalicious-MUY [Trj]
BayCheck package	Win32:Evo-gen [Susp]
DefaultTab package	Win32:Evo-gen [Susp]
Digistar package	Win32:Evo-gen [Susp]
FullCircle package	Win32:Malware-gen
Ikea package	Win32:Evo-gen [Susp]
Konica package	Win32:Evo-gen [Susp]
MPlus package	Other:Malware-gen [Trj]
MusicArena package	Win32:Evo-gen [Susp]
Nero package	Other:Malware-gen [Trj]
Nvidia package	Win32:Trojan-gen
PopUpWasher package	Win32:Evo-gen [Susp]
RadioTracker package	Win32:Evo-gen [Susp]
ServersCheck package	Win32:Evo-gen [Susp]
Sony package	Win32:Evo-gen [Susp]
SysReport package	Win32:Evo-gen [Susp]
vSkype package	Win32:Evo-gen [Susp]

 

Copyright and Disclaimer

This publication is Copyright © 2016 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.

For more information about AV-Comparatives and the testing methodologies, please visit our website.

AV-Comparatives
(April 2016)