Interpreting MITRE Engenuity ATT&CK Evaluations

By George Tubin, Director of Product Strategy, Cynet

Thorough, independent tests are a vital resource as cybersecurity leaders and their teams evaluate vendors’ abilities to guard against increasingly sophisticated threats to their organization. And perhaps no assessment is more widely trusted than the annual MITRE Engenuity ATT&CK Evaluations: Enterprise.

This evaluation is critical for testing vendors because it’s virtually impossible to evaluate cybersecurity vendors based on their own performance claims. Along with vendor reference checks and proof of value evaluations (POV) — a live trial — in their environment, the MITRE Engenuity results add additional objective input to holistically assess cybersecurity vendors.

In this piece, we’ll unpack MITRE Engenuity’s most recent methodology to test security vendors against real-world threats, offer our interpretation of the results and identify the top takeaways emerging from the evaluation of Cynet’s all-in-one security solution.

How Does MITRE Engenuity Test Vendors During the Evaluation?

The MITRE Engenuity ATT&CK Evaluation is performed by MITRE Engenuity and tests the endpoint protection solutions against a simulated attack sequence based on real-life approaches taken by well-known advanced persistent threat (APT) groups. The MITRE Engenuity ATT&CK Evaluations: Enterprise tested 29 vendor solutions by emulating the attack sequences of Turla, a sophisticated Russia-based threat group known to have infected victims in over 45 countries.

An important caveat is that MITRE does not rank or score vendor results. Instead, the raw test data is published along with some basic online comparison tools. Buyers then use that data to evaluate the vendors based on their organization’s unique priorities and needs. The participating vendors’ interpretations of the results are just that — their interpretations.

So, How Do You Interpret the Results?

That’s a great question — one that a lot of people are asking themselves right now. The MITRE Engenuity ATT&CK Evaluations: Enterprise results aren’t presented in a format that many of us are used to digesting (looking at you, magical graph with quadrants).

And independent researchers often declare “winners” to lighten the cognitive load of figuring out which vendors are the top performers. In this case, identifying the “best” vendor is subjective. Which, if you don’t know what to look for, can feel like a hassle if you’re already frustrated with trying to assess which security vendor is the right fit for your organization. With these disclaimers issued, let’s now review the results themselves to compare and contrast how participating vendors performed against Turla.

MITRE Engenuity ATT&CK Results Summary

The following tables present Cynet’s analysis and calculation of all vendor MITRE Engenuity ATT&CK Evaluations: Enterprise test results for the most important measurements: Overall Visibility, Detection Accuracy, and Overall Performance.  There are a lot of other ways to look at the MITRE results, but we consider these to be most indicative of a solution’s ability to detect threats.

Overall Visibility is the total number of attack steps detected across all 143 sub-steps. Cynet defines Detection Quality as the percentage of attack sub-steps that included “Analytic Detections – those that identify the tactic (why an activity may be happening) or technique (both why and how the technique is happening).

Additionally, it’s important to look at how each solution performed before the vendor adjusted configuration settings due to missing a threat.  MITRE allows vendors to reconfigure their systems to attempt to detect threats that they missed or to improve the information they supply for detection. In the real world we don’t have the luxury of reconfiguring our systems due to missed or poor detection, so the more realistic measure is detections before configuration changes are implemented.

How’d Cynet do?

Based on Cynet’s analysis, our team is proud of our performance against Turla in the 2023 MITRE Engenuity ATT&CK Evaluations: Enterprise, outperforming the majority of vendors in several key areas. Here are our top takeaways:

Cynet’s all-in-one security solution was a top performer when evaluating both visibility and detection quality. This analysis illustrates how well a solution does in detecting threats and providing the context necessary to make the detections actionable. Missed detections are an invitation for a breach, while poor quality detections create unnecessary work for security analysts or potentially cause the alert to be ignored, which again, is an invitation for a breach.

Cynet delivered 100% visibility and perfectly detected every one of the 143 attack steps using no configuration changes.  The following chart shows the percentage of detections across all 143 attack sub-steps before the vendors implemented configuration changes. Cynet performed as well as two very large, well-known, security companies despite being a fraction of their size and far better than some of the biggest names in cybersecurity. 

Cynet provided analytic coverage for 100% of the 143 attack steps using no configuration changes. The following chart shows the percentage of detections that contained important general, tactic or technique information across the 143 attack sub-steps, again before configuration changes were implemented. Cynet performed as well as Palo Alto Networks, a $115 billion publicly traded company with 50 times the number of employees, and far better than many established, publicly traded brands.

Still have Questions?

In this on-demand webinar, Cynet CTO Aviad Hasnis and ISMG SVP Editorial Tom Field review the most recent MITRE ATT&CK results and share expert advice for cybersecurity leaders to find the vendor that best fits the specific needs of their organization. They also unpack Cynet’s performance during the tests and identify opportunities ot advance your team’s goals with the all-in-one security solution.

Blog courtesy of Cynet. Author George Tubin, Director of Product Strategy, Cynet. Regularly contributed guest blogs are part of MSSP Alert’s sponsorship program. Read more Cynet news and guest blogs here.