Sample quality for the Malware Protection Test
The test set for Malware Protection Test consisted of about 38,000 samples. As we only use samples that have been analysed by our own in-house automated sandboxes, the quality of our sets is very high. Unlike some other testers, we only use malware in our tests, and do not include PUAs or other controversial software. What is malicious and what is “potentially unwanted” is sometimes debatable. We welcome feedback from vendors; however, the decision as to whether something can or cannot be classified as malware is ultimately up to us, even if our decisions may sometimes be regarded as imperfect.
In selecting samples for the test set, we focus on current, relevant malware and not on exotic or extinct samples. Therefore, high detection/protection rates are to be expected from the participating products.
As usual, AV-Comparatives gives each participating vendor the possibility to review their missed samples AFTER the test using AV-Comparatives’ Feedback System. We continuously verify our test set during the open feedback time. During the feedback process we removed a number of samples even before any vendors had disputed them.
Of 21 vendors tested in this Malware Protection Test, 9 vendors provided feedback. Please note that many of the tested products use a third-party engine, and so they may rely on the engine provider to take care of disputes. We received 38 disputes (an average of just under 2 per participating vendor, median value 1), of which 25 were accepted. All the files disputed by vendors in this test performed suspicious activities (they can be seen as greyware/risky). None of the files were corrupted or clearly benign. Nevertheless, the disputes had no impact on the rankings/awards for this test. Regarding our dispute process in general, we have found that sometimes a vendor will try to dispute a sample, claiming that it is clean, even though it is clearly malicious. In such cases we reject the vendor’s claim, having investigated each individual disputed sample carefully.
Services such as VirusTotal have a valuable role to play, but they cannot replace real testing, as they usually rely on static command line scanners (often without a connection to the vendor’s cloud services). In other words, this means, that a sample not detected by an online multiscanner service, might be detected by the same vendor’s endpoint security product. Equally, detection of a sample by an online scanning engine does not mean that the sample is necessarily suitable for testing, as it cannot guarantee that a file fulfils our criteria for being malicious, or that it will run on a particular test operating system.
Below we have shared the hashes of 20 samples that were disputed in the past (some of those disputes were accepted, some not); users are invited to vote or comment on VirusTotal as to whether they think that the samples are malicious or not (as stated by VirusTotal, please vote “only if you have good evidence for it”).