The difference between AV-Comparatives’ EPR Test and MITRE ATT&CK Engenuity
Both the AV-Comparatives EPR Test and MITRE Engenuity have their merits, each providing useful insights into endpoint security solutions. Understanding the differences between these two tests is essential for IT managers, CISOs, and other tech-savvy professionals looking to select endpoint security solutions that will effectively protect their environments.
Test Scenarios
While MITRE Engenuity evaluates techniques from a single attack chain, carried out by a preselected APT announced in advance of the assessment, AV-Comparatives’ EPR Test involves 50 separate attack scenarios from undisclosed APTs. AV-Comparatives refrains from disclosing the attack methods and techniques in advance, mirroring real-world scenarios. This approach aims to showcase a solution’s ability to prevent, detect, and remediate attacks while providing passive response to users.
Traditionally, MITRE Engenuity participants are aware of the adversary groups chosen for upcoming evaluations through Calls for Participation. However, the Call for Participation in Managed Services Round 2023-2024 represents a departure from this practice. This latest Call for Contribution also allows participants to potentially impact test complexity by submitting data about APT tactics, techniques, and procedures known only to them.
Protection Capabilities
Historically, MITRE assessed solutions in Detect-Only mode, examining product responses to individual techniques within attack chains. However, MITRE only began testing Protection scenarios, meaning the attack was blocked or disrupted, in Round 3 (2020-2021). On the other hand, AV-Comparatives has been dedicated to validating Protection capabilities since the test’s inception in 2020.
Product Settings and Real-World Implementation
MITRE Engenuity permits the use of customized product settings and allows vendors to list these configurations on dedicated product pages. While vendors might use highly specific settings to enhance test results, these settings may not be practical for real-world use due to potential false positives, performance issues, and alert fatigue for real-world EDR/XDR operators. AV-Comparatives maintains control over setting changes, reporting them in the test results, so as to better inform users.
Scoring and Performance Comparisons
MITRE Engenuity lacks a straightforward scoring system to compare products’ effectiveness against threats and lacks comprehensive incident telemetry. AV-Comparatives addresses this gap by introducing a simple comparison scoring system, aiding customers in evaluating product efficiencies. Additionally, AV-Comparatives introduced a Total Cost of Ownership metric for product comparison, providing better insight into the numbers. MITRE Engenuity participants all claim to be the winner at the end. In contrast, AV-Comparatives testing poses greater challenges, allowing participants to remain anonymous. Achieving certification signifies exceptional proficiency, even for Strong Challengers.
Operational Accuracy and False Positives
Unlike AV-Comparatives’ EPR-Test, the MITRE Engenuity assessment does not consider False Positive scenarios (operational accuracy). This approach, combined with the flexibility to modify product configurations, introduces a risk of misinterpreting final results. In contrast, AV-Comparatives’ EPR-Test assesses operational accuracy and emphasizes the importance of balancing false negatives and operational accuracy.
Test Schedules
MITRE Engenuity testing occurs over varying timeframes, with several months potentially separating evaluations. Those who join early may have the option to be tested later. In AV-Comparatives’ EPR-Test the solutions are tested almost in parallel.
Telemetry and Threat Hunting
In devising its Engenuity tests, MITRE employs telemetry, heavily relying on data interpretation skills to uncover insights. Manufacturers with knowledge of what to search for are better equipped to uncover valuable findings in the data. Alternatively, AV-Comparatives devises its EPR tests based on its research of attack scenarios its specialists are aware of and have analyzed in depth themselves and also trained by the vendors before the test in order to operate their solutions efficiently.
Conclusion
In conclusion, the AV-Comparatives EPR Test and MITRE Engenuity tests offer complementary insights, each with a unique focus. It is essential for IT managers and other cybersecurity professionals to understand the differences between these tests and weigh these factors when selecting endpoint security solutions.
