It’s that time of the year again! The MITRE Engenuity ATT&CK evaluation results are in and generating quite the buzz in the industry. And for good reason.
The MITRE Engenuity ATT&CK evaluation focuses on a tool’s ability to prevent and detect cyber attacker behaviors. Now in its fourth round of testing, it has become the de-facto standard for how security solutions perform against different advanced cyberattack scenarios.
It’s interesting to note that MITRE Engenuity does not declare any winners or make any rankings. Instead, they provide transparent and comprehensive assessments of how each solution addressed threat prevention and detection at each step of the process. This leaves it up to the security community to make their own analyses and draw their own conclusions as to which solution best fits their specific needs. This impartial approach is one of the reasons why the MITRE Engenuity evaluation has gained popularity in recent years—a fact that is confirmed by the large number of security vendors (30) who have chosen to participate in this fourth round of evaluations.
You are probably curious to see how each solution performed. But before you start diving into the results, it’s important to revisit the MITRE ATT&CK evaluation framework to get a good understanding of the evaluation process—how it works, key evaluation categories, and notable changes from past evaluations.
MITRE Engenuity ATT&CK: At a Glance
First, let’s take a quick look at the MITRE ATT&CK framework. “ATT&CK” stands for “Adversarial Tactics, Techniques, and Common Knowledge,” and serves as a common language describing adversary behavior. MITRE describes it as: “a knowledge base of cyber adversary behavior and taxonomy for adversarial actions across their lifecycle.” And by “cyber adversary behavior”, it refers to tactics and techniques based on real-world observations that provide insights into:
- Who are the typical adversaries and why they attack
- How an attack is planned out, in terms of tactics and techniques used
- The nature of the threat and how to communicate it and respond to it faster
- What defenses can be proactively built
Every year, security solutions are evaluated against the ATT&CK framework and different real-world attack scenarios. And each evaluation includes a number of attack steps (tactics) and substeps (i.e., implementations of a technique).
What You Need To Know About This Year’s Evaluation
This year, the attack scenarios comprised of 19 steps with 109 substeps, covering a wide range of tactics and techniques. There are several important things to note about this latest evaluation, including:
- The focus was on the data encryption for impact technique abused in ransomware attacks.
- The detection evaluation scoring changed. Vendors received credit for the detection with the highest level of context (analytic coverage) provided by the solution at each substep. For this part of the evaluation, the preventative capabilities of each solution were disabled so that the evaluation could focus on the detection test.
- Similar to last year, the evaluation included an optional protections-focused scenario to determine a vendor’s ability to block key techniques and tactics instead of just identifying them. For this protection test, the preventative capabilities of each solution were enabled.
These aspects are important to keep in mind as you start digging through the results and drawing your own conclusions. So, let’s take a closer look at each item:
Data Encrypted For Impact
The 2022 MITRE ATT&CK enterprise evaluations focus on two groups of threat actors: Wizard Spider and Sandworm. MITRE emulates these adversaries’ tactics and techniques, with a focus on how they’ve abused the Data Encrypted for Impact technique. For example, the former leveraged data encryption for their ransomware attacks, including the Ryuk malware, while the latter leveraged encryption for destruction of data, as in the global NotPetya attack.
Given the 105% increase year-over-year in ransomware attacks, the focus on a technique abused in ransomware attacks is spot on. To get a better understanding of the tactics and techniques or the sub-steps of the attack scenarios, you can take a look at this year’s ATT&CK matrix.
“Scoring” Based on High Versus Low Detection Context
The detection categories used in the Wizard Spider and Sandworm evaluation range are differentiated based on the level of context provided, from “N/A” or no context at all, to “Technique”, which enriches detections with insights into how the attack is being performed, providing the highest level of context.
Source: https://attackevals.mitre-engenuity.org/enterprise/wizard-spider-sandworm/detection-categories
|
These detection categories are then worked into the final evaluation, which looks at:
- Analytic coverage = the ratio of substeps with detection enriched with analytic logic (i.e., general, tactic, and technique data)
- Telemetry coverage = the ratio of substeps with detections displaying only minimally processed data (i.e., telemetry)
- Visibility = the proportion of substeps with either an analytic or a telemetry detection (i.e., total coverage)
This “scoring” structure allows end users to weigh the types of detections against their needs, determine what they value most in a detection, and choose accordingly.
Last year, vendors received one point for each type of coverage their solution was able to provide. But this year, one point was received only for the type of coverage with the highest level of context provided by the solution at each substep, even though the solution was able to provide lower-level context detections in parallel.
This change in scoring is important because it provides more clarity on what vendor has more analytic coverage. If, for example, a vendor has a high telemetry coverage rate, that is not necessarily a good sign, as it can mean that their analytics were not able to enrich the basic telemetry data.
Protection Evaluation
This test looks at the solutions’ ability to block key techniques and tactics, instead of just discovering and logging them. The protection categories are straightforward, meaning that for each threat detected, the solution would either be able to block it (in which case it would be marked as “Blocked”) or not (in which case, it would be marked as “none”); in cases where blocking is not relevant, for each solution the protection piece would be marked as “not applicable.”
Blocking threats in real time is often preferred to just detecting them, which is why considering this protection piece as part of the evaluation is so important.
How Can You Leverage the MITRE ATT&CK Evaluation?
According to the “2022 State of the Market: The New Market Landscape Report – Pushing MSP Security to the Next Level”, 42% of the targeted attacks MSPs are experiencing most frequently are ransomware-driven. If you are sharing this experience, then this year’s MITRE ATT&CK Evaluation should be of particular interest to you and your customers. The MITRE ATT&CK evaluation helps you get a better understanding not only of some of the most devastating threats out there such as ransomware, but also of the security tools at your disposal and how you can plan your security strategy.
Starting from your own needs and priorities, you can use the ATT&CK framework to:
- understand what quality detections look like
- identify evidence of ATT&CK techniques in your environment
- build security metrics relevant to your environment
- identify gaps in your current security stack
- find security solutions that are relevant to your environment, your customers’ environments, and current threat landscape
For MSPs using N-able EDR, the 2022 MITRE ATTACK evaluation results brought great news. N-able EDR is powered by SentinelOne, a solution that leads the latest MITRE ATT&CK Evaluation results with:
- 100% Protection: (9 of 9 MITRE ATT&CK tests)
- 100% Detection: (19 of 19 attack steps)
- 100% Real-time (0 Delays)1
- 99% Visibility: (108 of 109 attack sub-steps)2
- 99% Highest Analytic Coverage: (108 of 109 detections)3
One more thing to keep in mind: Linux played a key role in this year’s test and SentinelOne detected 19/19 techniques tested on all operating systems, including Linux. As Linux powers most cloud workloads, not only is SentinelOne effective at providing user endpoints with unparalleled protection, but cloud workloads as well. This is increasingly important, because as more MSPs and MSP customers move to the cloud, they need to do that with speed and confidence.
So, as you browse through the MITRE ATT&CK evaluation results, always compare them against your own needs and priorities. And don’t forget—we’re always here for you should you like to learn more about N-able EDR and how it can help you meet your plans. All you have to do is reach out.
Emma Nistor is senior product marketing manager, N-able EDR. Read more N-able guest blogs here. Regularly contributed guest blogs are part of MSSP Alert’s sponsorship program.
