False Alarm Test March 2017

Introduction

This report is an appendix to the Malware Protection Test March 2017 listing details about the discovered False Alarms.

In AV testing, it is important to measure not only detection capabilities but also reliability. One aspect of reliability is the ability to recognize clean files as such, and not to produce false alarms (false positives). No product is immune from false positives (FPs), but some produce more than others. False Positives Tests measure which programs do best in this respect. There is no complete collection of all legitimate files that exist, and so no “ultimate” test of FPs can be done. What can be done, and is reasonable, is to create and use a set of clean files which is independently collected. If, when using such a set, one product has e.g. 30 FPs and another only 5, it is likely that the first product is more prone to FPs than the other. It doesn’t mean the product with 5 FPs doesn’t have more than 5 FPs globally, but it is the relative number that is important.

Tested Products

• Adaware Antivirus Pro 12.0Build: 12.0.636.1167
• Avast Free Antivirus 17.2Build: 17.2.3419.0
• AVG Free AntiVirus 17.2Build: 17.2.3419.0
• Avira Antivirus Pro 15.0Build: 15.0.24.146
• Bitdefender Internet Security 21.0Build: 21.0.23.1101
• BullGuard Internet Security 17.0Build: 17.0.329.1
• CrowdStrike Falcon Prevent 3.0Build: 3.0.5113.0
• Emsisoft Anti-Malware 2017Build: 2017.2.0.7219
• eScan Corporate 360 14.0Build: 14.0.1400.1957
• ESET Internet Security 10.0Build: 10.0.390.0
• F-Secure SAFE 14.176Build: 14.176.101
• Fortinet FortiClient 5.4Build: 5.4.2.0860
• Kaspersky Internet Security 17.0Build: 17.0.0.611 (c)
• McAfee Internet Security 19.0Build: 19.0.3060
• Microsoft Defender 4.10Build: 4.10.14393.0
• Panda Free Antivirus 18.0Build: 18.00.00
• Seqrite Endpoint Security 17.0Build: 17.00.10.2.2.2
• Symantec Norton Security 22.9Build: 22.9.0.71
• Tencent PC Manager 12.1Build: 12.1.26375.901
• Trend Micro Internet Security 11.0Build: 11.0.1186
• VIPRE Internet Security Pro 9.3Build: 9.3.6.3

Test Procedure

In order to give more information to the users about the false alarms, we try to rate the prevalence of the false alarms. Files which were digitally signed are considered more important. Due to that, a file with e.g. prevalence “level 1” and a valid digital signature is upgraded to the next level (e.g. prevalence “level 2”). Files which according to several telemetry sources had zero prevalence have been provided to the vendors in order to fix them, but have also been removed from the set and were not counted as false alarms.

The prevalence is given in five categories and labeled with the following colors: 

Level		Presumed number of affected users	Comments
1		Probably fewer than hundred users	Individual cases, old or rarely used files, very low prevalence
2		Probably several hundreds of users	Initial distribution of such files was probably much higher, but current usage on actual systems is lower (despite its presence), that is why also well-known software may now affect / have only a prevalence of some hundreds or thousands of users.
3		Probably several thousands of users
4		Probably several tens of thousands (or more) of users
5		Probably several hundreds of thousands or millions of users	Such cases are likely to be seen much less frequently in a false alarm test done at a specific time, as such files are usually either whitelisted or would be noticed and fixed very fast.

Most false alarms will probably fall into the first two levels most of the time. In our opinion, anti-virus products should not have false alarms on any sort of clean files regardless of how many users are currently affected by them. While some AV vendors may play down the risk of false alarms and play up the risk of malware, we are not going to rate products based on what the supposed prevalence of false alarms is. We already allow a certain amount of false alarms (currently 10) inside our clean set before we start penalizing scores, and in our opinion products which produce a higher amount of false alarms are also more likely to produce false alarms on more prevalent files (or in other sets of clean files). The prevalence data we give about clean files is just for informational purpose. The listed prevalence can differ inside the report, depending on which file/version the false alarm occurred, and/or how many files of the same kind were affected.

Testcases

All listed false alarms were encountered at the time of testing. False alarms caused by unencrypted data blocks in anti-virus related files were not counted. If a product had several false alarms belonging to the same application, it is counted here as only one false alarm. Cracks, keygens, or other highly questionable tools, including FPs distributed/shared primarily by vendors (which may be in the several thousands) or other non-independent sources are not counted here as false positives.

Test Results

There may be a variation in the number of false positives produced by two different programs that use the same engine (principal detection component). For example, Vendor A may license its detection engine to Vendor B, but Vendor A’s product may have more or fewer false positives than Vendor B’s product. This can be due to factors such as different internal settings being implemented, differences in other components and services such as additional or differing secondary engines/signatures/whitelist databases/cloud services/quality assurance, and possible time delay between the release of the original signatures and the availability of the signatures for third-party products.

False Positives (FPs) are an important measurement for AV quality. One FP report from a customer can result in large amount of engineering and support work to resolve the issue.  Sometimes this can even lead to important data loss or system unavailability.  Even “not significant” FPs (or FPs on old applications) deserve mention and attention because FPs are likely to be a result of principled rule detections.  It just happened that the FP was on an insignificant file. The FP possibility is probably still in the product and could cause an FP again on a more significant file. Thus, they still deserve mention and still deserve to be penalised. Below you will find the false alarms we observed in our independent set of clean files. Red entries highlight false alarms on files that were digitally signed.

1.	ESET, Microsoft	0	no/very few FPs
2.	Panda	1
3.	Adaware, Avira, Bitdefender, eScan	2	few FPs
4.	BullGuard, Seqrite, Tencent, VIPRE	3
5.	Emsisoft, Fortinet, Kaspersky	4
6.	F-Secure	6
7.	McAfee	9
8.	Avast, AVG	20	many FPs
9.	Trend Micro	29
10.	Symantec	96	very many FPs
11.	CrowdStrike	125	remarkably many FPs

Details about the discovered false alarms

ESET and Microsoft had zero false alarms on the used set of clean files.

Panda		1 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
SafeNSec package	Trj/CI.A

 

Adaware		2 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
PDFexperte package	Gen:Variant.Symmi.71281
XFire package	Gen:Variant.Razy.95563

 

Bitdefender		2 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
PDFexperte package	Gen:Variant.Symmi.71281
XFire package	Gen:Variant.Razy.95563

 

eScan		2 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
PDFexperte package	Gen:Variant.Symmi.71281 (DB)
XFire package	Gen:Variant.Razy.95563 (DB)

 

AVIRA		2 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
AutoIt package	TR/SelfDel.hvov Trojan
Vuex package	HEUR/APC (Cloud)

 

BullGuard		2 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
PDFexperte package	Gen:Variant.Symmi.71281
SevenRemix package	Trojan.NSIS.Androm.3
XFire package	Gen:Variant.Razy.95563

 

Seqrite		3 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
PDFexperte package	EE:Malwr.Heur.Symmi.71281
SevenRemix package	EE:Trojan.NSIS.Androm.3
XFire package	EE:Malwr.Heur.Razy.95563

 

Tencent		3 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
PDFexperte package	Gen:Variant.Symmi.71281
SevenRemix package	Trojan.NSIS.Androm.3
XFire package	Gen:Variant.Razy.95563

 

VIPRE		3 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
PDFexperte package	Gen:Variant.Symmi.71281
SevenRemix package	Trojan.NSIS.Androm.3
XFire package	Gen:Variant.Razy.95563

 

Emsisoft		4 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
GXTranscoder package	Trojan.Generic.7464985 (B)
OrangeGem package	Trojan.Sinowal.Gen.1 (B)
PDFexperte package	Gen:Variant.Symmi.71281 (B)
XFire package	Gen:Variant.Razy.95563 (B)

 

Kaspersky Lab		4 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
AVG package	Trojan.Win32.Gofot.fch
Clipboard package	Trojan-Dropper.Win32.Dapato.olhj
FiveWin package	UDS:DangerousObject.Multi.Generic
KonfKit package	Trojan-Dropper.Win32.Agent.bjsczl

 

Fortinet		4 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
FileViewer package	PossibleThreat.SB
Matrox package	PossibleThreat.SB
PaperOffice package	W32/Injector.CVRZ!tr
Tram package	W32/Bouiben.OLE!tr

 

F-Secure		6 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
Frisk package	Trojan:W32/Gen4135.984577c56d!Online
GXTranscoder package	Trojan.Generic.7464985
OrangeGem package	Trojan.Sinowal.Gen.1
PDFexperte package	Gen:Variant.Symmi.71281
SevenRemix package	Trojan.NSIS.Androm.3
XFire package	Gen:Variant.Razy.95563

 

McAfee		9 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
ALZip package	Artemis!21095C3ECA72
CyberLink package	Artemis!45A71AA1FD0A
GameAccelerator package	Artemis!E05D6ECAC7EC
Komfort package	GenericRXAV-IV!87E29BAC562F
PCWizard package	Artemis!D9ED2981F363
Shanghai package	GenericRXAS-FB!6DF44A88D27C
TheSettlers package	Artemis!32C50B75BE89
ToolLoader package	Artemis!D80D89C8AFCB
Tris package	Artemis!1AB75BD08E97

 

Avast		20 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
AON package	Win32:DH-A1
Apfelman package	Win32:Malware-gen
ASUS package	Win32:DH-A1
Ausfuellen package	FileRepMalware
Browster package	Win32:DH-A1
Deenes package	Win32:DH-A1
EasyPAK package	Win32:Evo-gen
F-Secure package	Win32:DH-A1
GMER package	Win32:DH-A1
Google package	Win32:DH-A1
Horse package	Win32:Evo-gen
HP package	Win32:DH-A1
Kaffeepause package	Win32:DH-A1
Kicker package	Win32:DH-A1
NetInfo package	Win32:Evo-gen
OpenOffice package	Win32:Evo-gen
Puzzle package	Win32:DH-A1
SunBird package	Win32:Evo-gen
WinAmp package	Win32:Evo-gen
XPburn package	Win32:DH-A1

 

AVG		20 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
AON package	Win32:DH-A1
Apfelman package	Win32:Malware-gen
ASUS package	Win32:DH-A1
Ausfuellen package	FileRepMalware
Browster package	Win32:DH-A1
Deenes package	Win32:DH-A1
EasyPAK package	Win32:Evo-gen
F-Secure package	Win32:DH-A1
GMER package	Win32:DH-A1
Google package	Win32:DH-A1
Horse package	Win32:Evo-gen
HP package	Win32:DH-A1
Kaffeepause package	Win32:DH-A1
Kicker package	Win32:DH-A1
NetInfo package	Win32:Evo-gen
OpenOffice package	Win32:Evo-gen
Puzzle package	Win32:DH-A1
SunBird package	Win32:Evo-gen
WinAmp package	Win32:Evo-gen
XPburn package	Win32:DH-A1

 

Trend Micro		29 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
Adobe package	Suspicious File
Bluefish package	Suspicious File
Busch package	Suspicious File
ClearProg package	Suspicious File
ClikcMedia package	Suspicious File
Core2Max package	Suspicious File
Deeenes package	Suspicious File
FireStorm package	Suspicious File
FiveWin package	Suspicious File
Floppy package	Suspicious File
FreeSpa package	Suspicious File
Frisk package	Suspicious File
F-Secure package	Suspicious File
GMER package	Suspicious File
Grub2Win package	Suspicious File
GXTranscoder package	Suspicious File
IconHider package	Suspicious File
Kalenderpack package	Suspicious File
Opera package	Suspicious File
Puzzle package	Suspicious File
RegBrowser package	Suspicious File
RegCool package	Suspicious File
TennisMaster package	Suspicious File
ToolLoader package	Suspicious File
Tram package	Suspicious File
Tris package	Suspicious File
VistaMizer package	Suspicious File
XFire package	Suspicious File
Zup package	Suspicious File

 

Symantec Norton		96 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
3Dtweak package	see notes below
AAMS package	see notes below
ActivePixels package	see notes below
Amok package	see notes below
AppStarter package	see notes below
ASuite package	see notes below
AutoHotKey package	see notes below
AverTV package	see notes below
BOM package	see notes below
CallSearch package	see notes below
CamStudio package	see notes below
ClearProg package	see notes below
ClickClean package	see notes below
Clickster package	see notes below
Clipboard package	see notes below
CopyToDV package	see notes below
Daphne package	see notes below
DateiCommander package	see notes below
Datenabgleich package	see notes below
DisableWinTracking package	see notes below
Disconnect package	see notes below
DiskCleaner package	see notes below
DriveScan package	see notes below
DVdisaster package	see notes below
Ewido package	see notes below
FileSync package	see notes below
FinalBurner package	see notes below
FinePrint package	see notes below
Floppy package	see notes below
FreeCommander package	see notes below
FreshDow package	see notes below
FreshUI package	see notes below
GigaByte package	see notes below
GrabIt package	see notes below
GrandPrix package	see notes below
GXTranscoder package	see notes below
HTMLdoc package	see notes below
Hugin package	see notes below
HyperSnap package	see notes below
IconHider package	see notes below
Imagine package	see notes below
InfraRecorder package	see notes below
Jpegr package	see notes below
Kaspersky package	see notes below
Kismet package	see notes below
Koolader package	see notes below
MediaInfo package	see notes below
MegaCamManager package	see notes below
Megui package	see notes below
MouseButton package	see notes below
Mozilla package	see notes below
MultiStage package	see notes below
PeaZip package	see notes below
PhotoMatix package	see notes below
PingPlotter package	see notes below
Polarr package	see notes below
QuizPro package	see notes below
RainMeter package	see notes below
RegBrowser package	see notes below
RegCool package	see notes below
RegistyrFirstAid package	see notes below
RestoreNatur package	see notes below
Sandboxie package	see notes below
Screamer package	see notes below
ScreenshotCaptor package	see notes below
SevenRemix package	see notes below
Shutdown package	see notes below
SimplyZip package	see notes below
SmartMorph package	see notes below
Snowflakes package	see notes below
SSC package	see notes below
StationRipper package	see notes below
Sylpheed package	see notes below
Tagscan package	see notes below
TeraCopy package	see notes below
TheBestMovie package	see notes below
Tintii package	see notes below
TweakPower package	see notes below
USBwatcher package	see notes below
VHDattach package	see notes below
VideoInspector package	see notes below
VirtualMoon package	see notes below
VistaMizer package	see notes below
VistaServices package	see notes below
VisualTask package	see notes below
Vundofix package	see notes below
WinAmp package	see notes below
Winboard package	see notes below
WindowHiding package	see notes below
WinPlus package	see notes below
WinRAR package	see notes below
WiseDiskCleaner package	see notes below
XBbrowser package	see notes below
Xenu package	see notes below
XPprome package	see notes below
Zar package	see notes below

Symantec requested to remove from this report the detection names of their false alarms.
Symantec Norton had 96 false alarms. Symantec is a new re-entry in our tests – it is to be expected that their number of false alarms will be much lower next time.

CrowdStrike		125 False Alarm
False alarm found in some parts of	Detected as	Supposed prevalence
Acer package	High (Activity Prevented)
Adobe package	High (Activity Prevented)
AmericanConquest package	High (Activity Prevented)
AncientSpider package	High (Activity Prevented)
ASUS package	High (Activity Prevented)
AutoFlowChart package	High (Activity Prevented)
BigFish package	High (Activity Prevented)
BIMexplorer package	High (Activity Prevented)
BloodRayne package	High (Activity Prevented)
BlueFish package	High (Activity Prevented)
BrainTwister package	High (Activity Prevented)
Busch package	High (Activity Prevented)
CDSearch package	High (Activity Prevented)
ClickMedia package	High (Activity Prevented)
CloneDVD package	High (Activity Prevented)
Cobra11 package	High (Activity Prevented)
Compare package	High (Activity Prevented)
Comtest package	High (Activity Prevented)
Core2Max package	High (Activity Prevented)
Daemon package	High (Activity Prevented)
Databecker package	High (Activity Prevented)
DataNation package	High (Activity Prevented)
DeltaForce package	High (Activity Prevented)
DiaShow package	High (Activity Prevented)
DieSiedler package	High (Activity Prevented)
DTB package	High (Activity Prevented)
Duden package	High (Activity Prevented)
EasyBurning package	High (Activity Prevented)
EmailPro package	High (Activity Prevented)
eMedia package	High (Activity Prevented)
eRightSoft package	High (Activity Prevented)
ESET package	High (Activity Prevented)
ESreg package	High (Activity Prevented)
EuroRechner package	High (Activity Prevented)
FileTime package	High (Activity Prevented)
Findus package	High (Activity Prevented)
FireStorm package	High (Activity Prevented)
Flyakite package	High (Activity Prevented)
Fotofit package	High (Activity Prevented)
Fotograf package	High (Activity Prevented)
FreeCrypt package	High (Activity Prevented)
FreeSpa package	High (Activity Prevented)
FTCheck package	High (Activity Prevented)
Fujitsu package	High (Activity Prevented)
FunnyMemo package	High (Activity Prevented)
GoogleClean package	High (Activity Prevented)
Groowe package	High (Activity Prevented)
Grub2Win package	High (Activity Prevented)
Harrys package	High (Activity Prevented)
HDCleaner package	High (Activity Prevented)
Helper package	High (Activity Prevented)
Horseshoe package	High (Activity Prevented)
HP package	High (Activity Prevented)
HPV package	High (Activity Prevented)
IceAge package	High (Activity Prevented)
IconXtractor package	High (Activity Prevented)
IDman package	High (Activity Prevented)
IMG package	High (Activity Prevented)
ImperialGlory package	High (Activity Prevented)
InstallShield package	High (Activity Prevented)
JoWood package	High (Activity Prevented)
Kalenderpack package	High (Activity Prevented)
Kao package	High (Activity Prevented)
Keller package	High (Activity Prevented)
Kreed package	High (Activity Prevented)
Lexikon package	High (Activity Prevented)
Loop package	High (Activity Prevented)
MathCAD package	High (Activity Prevented)
Maulwurf package	High (Activity Prevented)
Miui package	High (Activity Prevented)
Moorhuhn package	High (Activity Prevented)
MPBrowser package	High (Activity Prevented)
MSI package	High (Activity Prevented)
MusicMaker package	High (Activity Prevented)
Nero package	High (Activity Prevented)
NetPing package	High (Activity Prevented)
NNG package	High (Activity Prevented)
Nokia package	High (Activity Prevented)
OOBE package	High (Activity Prevented)
Opera package	High (Activity Prevented)
Othello package	High (Activity Prevented)
OutlookTuner package	High (Activity Prevented)
PartyGaming package	High (Activity Prevented)
PCG package	High (Activity Prevented)
PerfWatch package	High (Activity Prevented)
Printy package	High (Activity Prevented)
Raetsel package	High (Activity Prevented)
RAMid package	High (Activity Prevented)
RegCool package	High (Activity Prevented)
RegSnap package	High (Activity Prevented)
RegToy package	High (Activity Prevented)
RSW package	High (Activity Prevented)
Sidebar package	High (Activity Prevented)
Simkl package	High (Activity Prevented)
SkiRacing package	High (Activity Prevented)
StartDelay package	High (Activity Prevented)
StyleSelector package	High (Activity Prevented)
Sudoku package	High (Activity Prevented)
TagesAnzeiger package	High (Activity Prevented)
Tankwart package	High (Activity Prevented)
TennisMasters package	High (Activity Prevented)
Texiterator package	High (Activity Prevented)
Tierpension package	High (Activity Prevented)
TrackWin package	High (Activity Prevented)
TrendMicro package	High (Activity Prevented)
TrueCrypt package	High (Activity Prevented)
TweakNow package	High (Activity Prevented)
TweakPower package	High (Activity Prevented)
UltimateSudoku package	High (Activity Prevented)
Uniextract package	High (Activity Prevented)
Unitype package	High (Activity Prevented)
Unreal package	High (Activity Prevented)
USBagent package	High (Activity Prevented)
Wanywaord package	High (Activity Prevented)
Warcraft package	High (Activity Prevented)
WesternDigital package	High (Activity Prevented)
WildlifePark package	High (Activity Prevented)
Wildtangent package	High (Activity Prevented)
WinAmp package	High (Activity Prevented)
WinDVD package	High (Activity Prevented)
WinSecurityGate package	High (Activity Prevented)
Withopf package	High (Activity Prevented)
Wobram package	High (Activity Prevented)
Zup package	High (Activity Prevented)
zWetter package	High (Activity Prevented)

CrowdStrike had 125 false alarms. Please note that CrowdStrike Falcon Prevent is a product made for corporate users. Many of the FP’s encountered in this test would not cause a big problem in such an environment; for example, several of them are game-related files which would not be allowed in a corporate network anyway. CrowdStrike is also a new entry in our tests – it is to be expected that their number of false alarms will be much lower next time.

Copyright and Disclaimer

This publication is Copyright © 2017 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.

For more information about AV-Comparatives and the testing methodologies, please visit our website.

AV-Comparatives
(April 2017)