This website uses cookies to ensure you get the best experience on our website.
Please note that by continuing to use this site you consent to the terms of our Privacy and Data Protection Policy.
Accept

False Alarm Test March 2016

Date March 2016
Language English
Last Revision April 14th 2016

Appendix to the Anti-Virus Comparatives March 2016


Release date 2016-04-15
Revision date 2016-04-14
Test Period March 2016
Online with cloud connectivity checkbox-checked
Update allowed checkbox-checked
False Alarm Test included checkbox-unchecked
Platform/OS Microsoft Windows

Introduction

This report is an appendix to the File Detection Test September 2016 listing details about the discovered False Alarms.

With AV testing it is important to measure not only detection capabilities but also reliability. One aspect of reliability is the ability to recognize clean files as such, and not produce false alarms (false positives). No product is immune from false positives (FPs), but some produce more than others, and the our goal is to find out which programs do best in this respect. There is no complete collection of all legitimate files that exist, and so no “ultimate” test of FPs can be done. What can be done, and is reasonable, is to create and use a set of clean files which is independently collected. If with such a set one product has e.g. 30 FPs and another only 5, it is likely that the first product is more prone to FP’s than the other. It doesn’t mean the product with 5 FPs doesn’t have more than 5 FPs globally, but it is the relative number that is important.

Tested Products

Test Procedure

In order to give more information to the users about the false alarms, we try to rate the prevalence of the false alarms. Files which were digitally signed are considered more important. Due to that, a file with e.g. prevalence “level 1” and a valid digital signature is upgraded to the next level (e.g. prevalence “level 2”). Files which according to several telemetry sources had zero prevalence have been provided to the vendors in order to fix them, but have also been removed from the set and were not counted as false alarms.

The prevalence is given in five categories and labeled with the following colors: fp_prevalence

LevelPresumed number of affected usersComments
1fp_prevalence_1Probably fewer than hundred usersIndividual cases, old or rarely used files, unknown prevalence
2fp_prevalence_2Probably several hundreds of usersInitial distribution of such files was probably much higher, but current usage on actual systems is lower (despite its presence), that is why also well-known software may now affect / have only a prevalence of some hundreds or thousands of users.
3fp_prevalence_3Probably several thousands of users
4fp_prevalence_4Probably several tens of thousands (or more) of users
5fp_prevalence_5Probably several hundreds of thousands or millions of usersSuch cases are likely to be seen much less frequently in a false alarm test done at a specific time, as such files are usually either whitelisted or would be noticed and fixed very fast.

Most false alarms will probably fall into the first two levels most of the time. In our opinion, anti-virus products should not have false alarms on any sort of clean files regardless of how many users are currently affected by them. While some AV vendors may play down the risk of false alarms and play up the risk of malware, we are not going to rate products based on what the supposed prevalence of false alarms is. We already allow a certain amount of false alarms (currently 10) inside our clean set before we start penalizing scores, and in our opinion products which produce a higher amount of false alarms are also more likely to produce false alarms on more prevalent files (or in other sets of clean files). The prevalence data we give about clean files is just for informational purpose. The listed prevalence can differ inside the report, depending on which file/version the false alarm occurred, and/or how many files of the same kind were affected.

Testcases

All listed false alarms were encountered at the time of testing. False alarms caused by unencrypted data blocks in anti-virus related files were not counted. If a product had several false alarms belonging to the same software, it is counted here as only one false alarm. Cracks, keygens, etc. or other highly questionable tools, including FPs distributed/shared primarily by vendors (which may be in the several thousands) or other non-independent sources are not counted here as false positives.

Test Results

Some products using third-party engines/signatures may have fewer or more false alarms than the licensed engine has by its own, e.g. due to different internal settings implemented, the additional checks/engines/clouds/signatures, whitelist databases, time delay between the release of the original signatures and the availability of the signatures for third-party products, additional quality assurance of signatures before release, etc.

False Positives (FPs) are an important measurement for AV quality.  One FP report from a customer can result in large amount of engineering and support work to resolve the issue.  Sometimes this can even lead to important data loss or system unavailability.  Even “not significant” FPs (or FPs on old applications) deserve mention and attention because FPs are likely to be a result of principled rule detections.  It just happened that the FP was on an insignificant file. The FP possibility is probably still in the product and could cause an FP again on a more significant file. Thus, they still deserve mention and still deserve to be penalised. Below you will find the false alarms we observed in our independent set of clean files. Red entries highlight false alarms on files that were digitally signed.

1.ESET, Trend Micro0very few FPs
2.McAfee1 few FPs
3.BullGuard, eScan, Sophos2
4.Bitdefender, Emsisoft, Kaspersky Lab, ThreatTrack3
5.F-Secure, Lavasoft, Tencent4
6.Quick Heal8
7.Avira9 many FPs
8.AVG10
9.Fortinet, Microsoft13
10.Avast17

Details about the discovered false alarms

ESET and Trend Micro had zero false alarms on the used set of clean files.

McAfee 1 False Alarm
False alarm found in some parts of Detected as Supposed prevalence
TVgenial package Artemis!E1DB26418B72 fp_prevalence_4

 

BullGuard 2 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Granny package Gen:Variant.Razy.19282 fp_prevalence_5
Runner package Gen:Variant.Barys.49628  fp_prevalence_4

 

eScan 2 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Corel package Gen:Variant.Barys.52348 (DB) fp_prevalence_1
Runner package Gen:Variant.Barys.49628 (DB) fp_prevalence_4

 

Sophos 2 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
BZIP package Mal/Dorf-D fp_prevalence_5
TNI package Mal/Generic-L fp_prevalence_1

 

Bitdefender 3 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Corel package Gen:Variant.Barys.52348 fp_prevalence_1
Granny package Gen:Variant.Razy.19282 fp_prevalence_5
Runner package Gen:Variant.Barys.49628 fp_prevalence_4

 

Emsisoft 3 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Corel package Gen:Variant.Barys.52348 (B) fp_prevalence_1
Granny package Gen:Variant.Razy.19282 (B) fp_prevalence_5
Runner package Gen:Variant.Barys.49628 (B) fp_prevalence_4

 

Kaspersky Lab 3 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
OnlineEye package Trojan-Downloader.Win32.Banload.aajbo  fp_prevalence_1
Puzzle package Trojan-Spy.Win32.Taopap.phe fp_prevalence_1
Radeon package P2P-Worm.Win32.Palevo.hynv fp_prevalence_3

 

ThreatTrack 3 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Corel package Gen:Variant.Barys.52348 fp_prevalence_1
Granny package Gen:Variant.Razy.19282 fp_prevalence_5
Runner package Gen:Variant.Barys.49628 fp_prevalence_4

 

F-Secure 4 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Corel package Gen:Variant.Barys.52348 fp_prevalence_1
FinePrint package Trojan:W32/Gen4135.1fc23018e8!Online fp_prevalence_3
Runner package Gen:Variant.Barys.49628 fp_prevalence_4
Xtreme package Trojan-dropper:W32/Coinminer.99db20ce3c!Online fp_prevalence_3

 

Lavasoft 4 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Corel package Gen:Variant.Barys.52348 fp_prevalence_1
Granny package Gen:Variant.Razy.19282 fp_prevalence_5
Mame package Gen:Variant.Barys.52421 fp_prevalence_3
Runner package Gen:Variant.Barys.49628 fp_prevalence_4

 

Tencent 4 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Corel package Gen:Variant.Barys.52348 fp_prevalence_1
Granny package Gen:Variant.Razy.19282 fp_prevalence_5
Mame package Gen:Variant.Barys.52421 fp_prevalence_3
Runner package Gen:Variant.Barys.49628 fp_prevalence_4

 

Quick Heal 8 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Elsword package Trojanspy.Agent.018127 fp_prevalence_5
Granny package EE:Malwr.Heur.Razy.19282 fp_prevalence_5
IronBrowser package JS/Agent.KK fp_prevalence_1
MakeDisk package Ransom.Crowti.A4 fp_prevalence_4
PerfectMenu package Trojan.Malagent.019169 fp_prevalence_3
Runner package EE:Malwr.Heur.Barys.49628 fp_prevalence_4
Screensaver package Trojan.Scar.013919 fp_prevalence_5
WB package Suspicious fp_prevalence_4

 

AVIRA 9 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
AudaPad package HEUR/APC fp_prevalence_4
Chilkat package HEUR/APC fp_prevalence_1
CreateAMall package HEUR/APC fp_prevalence_1
CueMaker package HEUR/APC fp_prevalence_3
Drei package HEUR/APC fp_prevalence_1
Fujitsu package HEUR/APC fp_prevalence_2
PlantsVSZombies package HEUR/APC fp_prevalence_1
Tiscali package HEUR/APC fp_prevalence_3
WinHotel package HEUR/APC fp_prevalence_2

 

AVG 10 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Acer package Zbot.AJKR fp_prevalence_5
AirSnare package Collected_c.CGRB fp_prevalence_3
ArrowSearch package Win32/DH{d4IRgQw} fp_prevalence_3
DigitalTheatre package Win32/DH{cjETMHmBRg?} fp_prevalence_5
MightyChicken package Win32/DH{gVGBCoFT?} fp_prevalence_3
PowerTranslator package Win32/DH{ZzWCHIEPgRxB?} fp_prevalence_4
SIW package Generic36.CGMO fp_prevalence_3
SysTrayX package Agent5.AKKG fp_prevalence_1
VirtualExpander package Win32/DH{gg92A1g?} fp_prevalence_5
Zattoo package Win32/Herz fp_prevalence_4

 

Fortinet 13 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
ASUS package W32/Agent.NESVWS!tr fp_prevalence_3
CableMon package W32/Generic.AC.2181457 fp_prevalence_3
ColdFusion package W32/Generic.AC.2506367 fp_prevalence_4
HWinfo package W32/Bayrob.AT!tr fp_prevalence_4
Macromedia package W32/Generic.AC.2506367 fp_prevalence_3
PageDfrg package PossibleThreat.SB!tr.rkit fp_prevalence_4
Pi package W32/Kryptik.EKOM!tr fp_prevalence_5
SkinPack package W32/Sim.SP!tr fp_prevalence_3
Startupo package W32/Generic.AC.256673 fp_prevalence_2
SysOpt package INF/Qhost!tr fp_prevalence_1
Triton package W32/Generic.AC.2926293 fp_prevalence_5
WireShark package W32/Kryptik.EMEK!tr fp_prevalence_2
WS_FTP package W32/Kryptik.ELYI!tr fp_prevalence_1

 

Microsoft 13 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
2H4U package Trojan:Win32/Varpes.J!plock fp_prevalence_3
Battlefield package Trojan:Win32/Skeeyah.A!bit fp_prevalence_3
ClipInc package Trojan:Win32/Dorv.C!rfn fp_prevalence_3
Dbox package Trojan:Win32/Varpes.J!plock fp_prevalence_1
DerLauncher package Trojan:Win32/Varpes.K!plock fp_prevalence_2
Fotokasten package Trojan:Win32/Varpes.J!plock fp_prevalence_3
HiddenFinder package Trojan:Win32/Varpes.K!plock fp_prevalence_1
KeriverImage package Trojan:Win32/Varpes.J!plock fp_prevalence_1
MediaCenter package Trojan:Win32/Varpes.J!plock fp_prevalence_1
MoviePlus package Trojan:Win32/Varpes.K!plock fp_prevalence_2
Nero package Trojan:Win32/Varpes.J!plock fp_prevalence_3
OrgaMax package Trojan:Win32/Varpes.J!plock fp_prevalence_1
Outlookers package Trojan:Win32/Varpes.J!plock fp_prevalence_1

 

Avast 17 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Adobe package Win32:GenMalicious-MUY [Trj] fp_prevalence_5
BayCheck package Win32:Evo-gen [Susp] fp_prevalence_1
DefaultTab package Win32:Evo-gen [Susp] fp_prevalence_5
Digistar package Win32:Evo-gen [Susp] fp_prevalence_1
FullCircle package Win32:Malware-gen fp_prevalence_1
Ikea package Win32:Evo-gen [Susp] fp_prevalence_2
Konica package Win32:Evo-gen [Susp] fp_prevalence_4
MPlus package Other:Malware-gen [Trj] fp_prevalence_3
MusicArena package Win32:Evo-gen [Susp] fp_prevalence_1
Nero package Other:Malware-gen [Trj] fp_prevalence_4
Nvidia package Win32:Trojan-gen fp_prevalence_1
PopUpWasher package Win32:Evo-gen [Susp] fp_prevalence_1
RadioTracker package Win32:Evo-gen [Susp] fp_prevalence_1
ServersCheck package Win32:Evo-gen [Susp] fp_prevalence_1
Sony package Win32:Evo-gen [Susp] fp_prevalence_5
SysReport package Win32:Evo-gen [Susp] fp_prevalence_3
vSkype package Win32:Evo-gen [Susp] fp_prevalence_1

 

Copyright and Disclaimer

This publication is Copyright © 2016 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.

For more information about AV-Comparatives and the testing methodologies, please visit our website.

AV-Comparatives
(April 2016)