This website uses cookies to ensure you get the best experience on our website.
Please note that by continuing to use this site you consent to the terms of our Privacy and Data Protection Policy.
Accept

False Alarm Test September 2021

Date September 2021
Language English
Last Revision October 10th 2021

Appendix to the Malware Protection Test September 2021


Release date 2021-10-15
Revision date 2021-10-10
Test Period September 2021
Online with cloud connectivity checkbox-checked
Update allowed checkbox-checked
False Alarm Test included checkbox-checked
Platform/OS Microsoft Windows

Introduction

This report is an appendix to the Malware Protection Test September 2021 listing details about the discovered False Alarms.

In AV testing, it is important to measure not only detection capabilities but also reliability. One aspect of reliability is the ability to recognize clean files as such, and not to produce false alarms (false positives). No product is immune from false positives (FPs), but some produce more than others. False Positives Tests measure which programs do best in this respect, i.e. distinguish clean files from malicious files, despite their context. There is no complete collection of all legitimate files that exist, and so no “ultimate” test of FPs can be done. What can be done, and is reasonable, is to create and use a set of clean files which is independently collected. If, when using such a set, one product has e.g. 15 FPs and another only 2, it is likely that the first product is more prone to FPs than the other. It doesn’t mean the product with 2 FPs doesn’t have more than 2 FPs globally, but it is the relative number that is important.

Tested Products

Test Procedure

In order to give more information to the user about the false alarms, we try to rate the prevalence of the false alarms. Files which were digitally signed are considered more important. Due to that, a file with the lowest prevalence level (Level 1) and a valid digital signature is upgraded to the next level (e.g. prevalence “Level 2”). Extinct files which according to several telemetry sources had zero prevalence have been provided to the vendors in order to fix them, but have also been removed from the set and were not counted as false alarms.

The prevalence is given in five categories and labeled with the following colors:fp_prevalence

LevelPresumed number of affected usersComments
1fp_prevalence_1Probably fewer than hundred usersIndividual cases, old or rarely used files, unknown prevalence
2fp_prevalence_2Probably several hundreds of usersInitial distribution of such files was probably much higher, but current usage on actual systems is lower (despite its presence), that is why also well-known software may now affect / have only a prevalence of some hundreds or thousands of users.
3fp_prevalence_3Probably several thousands of users
4fp_prevalence_4Probably several tens of thousands (or more) of users
5fp_prevalence_5Probably several hundreds of thousands or millions of usersSuch cases are likely to be seen much less frequently in a false alarm test done at a specific time, as such files are usually either whitelisted or would be noticed and fixed very fast.

Most false alarms will probably (hopefully) fall into the first two levels most of the time.

In our opinion, anti-virus products should not have false alarms on any sort of clean files regardless of how many users are currently affected by them. While some AV vendors may play down the risk of false alarms and play up the risk of malware, we are not going to rate products based on what the supposed prevalence of false alarms is. We already allow a certain number of false alarms (currently 10) inside our clean set before we start penalizing scores, and in our opinion products which produce a higher number of false alarms are also more likely to produce false alarms with more prevalent files (or in other sets of clean files). The prevalence data we give for clean files is just for informational purpose. The listed prevalence can differ inside the report, depending on which file/version the false alarm occurred, and/or how many files of the same kind were affected.

Testcases

All listed false alarms were encountered at the time of testing. False alarms caused by unencrypted data blocks in anti-virus related files were not counted. If a product had several false alarms belonging to the same application, it is counted here as only one false alarm. Cracks, keygens, or other highly questionable tools, including FPs distributed/shared primarily by vendors (which may be in the several thousands) or other non-independent sources are not counted here as false positives.

Test Results

There may be a variation in the number of false positives produced by two different programs that use the same engine (principal detection component). For example, Vendor A may license its detection engine to Vendor B, but Vendor A’s product may have more or fewer false positives than Vendor B’s product. This can be due to factors such as different internal settings being implemented, differences in other components and services such as additional or differing secondary engines/signatures/whitelist databases/cloud services/quality assurance, and possible time delay between the release of the original signatures and the availability of the signatures for third-party products.

False Positives (FPs) are an important measurement for AV quality. Furthermore, the test is useful and needed to avoid that vendors optimize products to score good in tests by looking at the context – this is why false alarms are being mixed and tested the same way as tests with malware are done. One FP report from a customer can result in large amount of engineering and support work to resolve the issue. Sometimes this can even lead to important data loss or system unavailability. Even “not significant” FPs (or FPs on older applications) deserve mention and attention because FPs are likely to be a result of principled rule detections. It just happened that the FP was on an insignificant file. The FP possibility is probably still in the product and could potentially cause an FP again on a more significant file. Thus, they still deserve mention and still deserve to be penalised. Below you will find some info about the false alarms we observed in our independent set of clean files. Red entries highlight false alarms on files that were digitally signed.

The detection names shown were taken mostly from pre-execution scan logs (where available). If a threat was blocked on/during/after execution (or no clear detection name was seen), we state “Blocked” in the column “Detected as”.

1.ESET, Microsoft1very few FPs
2.Avast, AVG, Kaspersky2 few FPs
3.TotalAV, Trend Micro3
4.Avira, McAfee, Total Defense4
5.G DATA, Malwarebytes8
6.Bitdefender, VIPRE9
7.NortonLifeLock14 many FPs
8.K756 very many FPs
9.Panda153 remarkably many FPs

Details about the discovered false alarms

 
ESET 1 False Alarm
False alarm found in some parts of Detected as Supposed prevalence
PCDefense package Win32/Agobot.NZU trojan

 

 
Microsoft 1 False Alarm
False alarm found in some parts of Detected as Supposed prevalence
Folder package Blocked

 

 
Avast 2 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
DotNet package IDP.Generic
Howard package AntiRansomware Shield (UD)

 

 
AVG 2 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
DotNet package IDP.Generic
Howard package Folder Protection

 

 
Kaspersky 2 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
CheckMail package HEUR:Worm.Win32.Generic
Faronics package HEUR:Trojan.Win32.Generic

 

 
Total AV 3 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Print package HEUR/APC
SlunkCrypt package HEUR/APC
VC package HEUR/AGEN
 
 
Trend Micro 3 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Password package Blocked
RegCool package Spyware
WSA package Blocked
 
 
Avira 4 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Print package HEUR/APC
Python HEUR/AGEN
SlunkCrypt package HEUR/APC
VC package HEUR/AGEN.1135086

 

 
McAfee 4 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
BreakQuest package Blocked
Calendar package Blocked
Folder package Blocked
Kid package Blocked

 

 
Total Defense 4 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Crillion package Blocked
SlunkCrypt package Gen:Variant.Razy.600253
UltimateFileManager package Gen:Variant.Strictor.199679
VirtualPiano package Gen:Variant.Strictor.258262

 

 
G Data 8 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
CFOS package Win32.Trojan.PSE.D9JJXP
Crillion package Blocked
DVDburning package Blocked
InstantPlayer package Blocked
RegSnap package Win32.Trojan.PSE.15LVZSU
SlunkCrypt package Gen:Variant.Razy.600253
UltimateFileManager Gen:Variant.Strictor.199679
VirtualPiano Gen:Variant.Strictor.258262

 

 
Malwarebytes 8 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
ETAC package Trojan.Agent.Gen
Extract package Trojan.Dropper.SFX
Pages package Blocked
Phed package Blocked
Skater package Blocked
SmadAV package Malware.Al.4015150073
WA package Blocked
YouTube package Blocked

 

 
Bitdefender 9 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Barcode package Blocked
Candy package Blocked
Crillion package Blocked
Page package Blocked
Preme package Blocked
SlunkCrypt package Gen.Variant.Razy.600253
UltimateFileManager Gen.Variant.Strictor.199679
Updater package Blocked
VirtualPiano Gen.Variant.Strictor.258262

 

 
VIPRE 9 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Barcode package Malware (General)
Candy package Malware (General)
Crillion package Virus.Generic
Page package Malware (General)
Preme package Malware (General)
SlunkCrypt package Virus.Generic
UltimateFileManager package Virus.Generic
Updater package Malware (General)
VirtualPiano package Virus.Generic

 

 
NortonLifeLock 14 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Adress package Blocked
AutoMKV package Heur.AdvML.C
Bricks package Trojan.Gen.2
CleanDisk package Heur.AdvML.C
Crillion package Heur.AdvML.C
Dimio package Heur.AdvML.B
DirectX package Trojan.Gen.X
EASO package Trojan.Gen
Listen package Blocked
MKV package Heur.AdvML.M
Moorhunt package Heur.AdvML.B
Privacy package Heur.AdvML.C
Pyth package Heur.AdvML.B
Zix package Heur.AdvML.B

 

 
K7 56 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
A43 package Blocked
Accent package Blocked
Advanced package Blocked
Apfelmann package Blocked
ArchiCrypt package Blocked
Audigy package Blocked
Brokers package Riskware
Cactus package Blocked
Canon package Riskware
Cleanit package Blocked
Corel package Riskware
DateiKatalog package Blocked
DigiVox package Riskware
DotNet package Riskware
Driver package Blocked
DVDburning package Blocked
EFmanager package Blocked
Feratel package Blocked
FileAnalyser package Blocked
FLV package Riskware
FreeView package Blocked
HDcleaner package Blocked
Ikaros package Blocked
Leadtech package Blocked
Listings package Blocked
Live package Blocked
MCM package Blocked
Moorhunt package Blocked
Multimedia package Blocked
Newsleecher package Blocked
OKI package Blocked
Optimize package Riskware
Optimizer package Riskware
Portable package Blocked
Preispiraten package Blocked
Preme package Blocked
Purge package Blocked
QCP package Blocked
Rainlendar package Blocked
Robot package Blocked
Safety package Blocked
Screenshot package Blocked
SmadAV package Riskware
Smarter package Riskware
Snow package Blocked
Spam package Blocked
Stacks package Blocked
StartTime package Blocked
StreamRipper package Blocked
SuperMario package Blocked
Symantec package Riskware
VanDerLee package Riskware
WinAmp package Blocked
Works package Riskware
Worms package Riskware
Zyxel package Blocked

 

 
Panda 153 False Alarms
False alarm found in some parts of Detected as Supposed prevalence
Addressbar package Blocked
Adress package Blocked
Agent package Blocked
Amazon package Blocked
AmericanConquest package Blocked
AnyVideo package Blocked
Arizona package Blocked
ATV package Blocked
AudioSplit package Blocked
Barcode package Blocked
BBL package Blocked
BCX package Blocked
Bitdefender package Blocked
Black package Blocked
BMC package Blocked
Boehm package Trj/Genetic.gen
Borsa package Blocked
Bricks package Blocked
BTC package Blocked
BTM package Blocked
ButtonShop package Blocked
Calculator package Blocked
Call package Blocked
Candy package Blocked
CareUEyes package Blocked
CFOS package Blocked
CheckIf package Blocked
CheckMail package Blocked
Clipboard package Blocked
Convert package Blocked
Corel package Blocked
Crillion package Blocked
Database package Blocked
DataRecovery package Blocked
Developers package Blocked
DFC package Blocked
Die package Blocked
DigiVox package Blocked
DirectX package Blocked
DirSaver package Blocked
Disable package Blocked
DpZip package Blocked
DrPhony package Blocked
DVDAuthor package Blocked
Earth package Blocked
EASO package Blocked
eBlinkx package Blocked
Editor package Blocked
Elevator package Blocked
Emma package Blocked
EnWeb package Blocked
Facebook package Blocked
FastFolders package Blocked
Feratel package Blocked
Firewall package Blocked
FlipPDF package Blocked
Floola package Blocked
Forms package Blocked
FoxIt package Blocked
Fraps package Blocked
FritzBox package Blocked
GAM package Blocked
Garrys package Blocked
GMST package Blocked
GPS package Blocked
Grafica package Blocked
Groowe package Blocked
Grub package Blocked
Howard package Blocked
Image package Blocked
Import package Blocked
Invadazoid package Blocked
Inventory package Blocked
Jukebox package Blocked
Kakao package Blocked
Kid package Blocked
KSA package Blocked
Labs package Blocked
LangManager package Blocked
Libertix package Blocked
License package Blocked
Lift package Blocked
Linq package Blocked
Logical package Blocked
Logyx package Blocked
Maintenance package Blocked
Makarevich package Blocked
Manager package Blocked
MP4 package Blocked
MTG package Blocked
MusicPlayer package Blocked
MyCar package Blocked
Netx package Blocked
NHsystems package Blocked
NorthStar package Blocked
Office package Blocked
OFX package Blocked
Outlook package Blocked
Page package Blocked
Password package Blocked
PDF2Video package Blocked
PDFCreator package Blocked
PersonalBackup package Blocked
Photo package Blocked
Pinner package Blocked
Pivi package Blocked
Player package Blocked
PRG package Blocked
Print package Blocked
Privacy package Blocked
QT package Blocked
QuickConvert package Blocked
RegSnap package Blocked
Reminder package Blocked
Remote package Blocked
RHF package Blocked
Ripper package Blocked
Robot package Blocked
Screen package Blocked
ScreenRecorder package Blocked
SDI package Blocked
Sim package Blocked
Simple package Blocked
SipGate package Blocked
Snapshot package Blocked
Solicad package Blocked
Sound package Blocked
Spamihilator package Blocked
Squirrel package Blocked
StatusIndicator package Blocked
Strapper package Blocked
Studio package Blocked
Sunbird package Blocked
SuperString package Blocked
Syntax package Blocked
System package Blocked
Tags package Blocked
TFT package Blocked
Toolkit package Blocked
Touch package Blocked
TubeMate package Blocked
Tweet package Blocked
UltimateFileManager package Blocked
Unity package Blocked
Updater package Blocked
Various package Blocked
Wave package Blocked
WinAmp package Blocked
Window package Blocked
WLAN package Blocked
Work package Blocked
WSUS package Blocked
YTD package Blocked

 

Copyright and Disclaimer

This publication is Copyright © 2021 by AV-Comparatives ®. Any use of the results, etc. in whole or in part, is ONLY permitted after the explicit written agreement of the management board of AV-Comparatives prior to any publication. AV-Comparatives and its testers cannot be held liable for any damage or loss, which might occur as result of, or in connection with, the use of the information provided in this paper. We take every possible care to ensure the correctness of the basic data, but a liability for the correctness of the test results cannot be taken by any representative of AV-Comparatives. We do not give any guarantee of the correctness, completeness, or suitability for a specific purpose of any of the information/content provided at any given time. No one else involved in creating, producing or delivering test results shall be liable for any indirect, special or consequential damage, or loss of profits, arising out of, or related to, the use or inability to use, the services provided by the website, test documents or any related data.

For more information about AV-Comparatives and the testing methodologies, please visit our website.

AV-Comparatives
(October 2021)