At a time when cyberattacks are getting more frequent and complex, when the cost of an attack is skyrocketing, and when hackers are well-funded and armed such advanced technologies as generative AI, running tests that gauge the ability of cybersecurity vendors to detect, respond to, and defend against them makes sense.
That’s what MITRE does through its ATT&CK Evaluation, with the results from the latest tests – round six – being released this week. In it, 19 vendors were pitted against two well-known ransomware strains in LockBit and Clop as well as malware that North Korean-linked threat groups run against macOS devices.
In all, there were three distinct attacks that included 16 steps and 80 sub-steps.
In the latest multi-step evaluations, the nonprofit – which also operates MITRE ATT&CK, a framework and knowledge base for modeling, detecting, and fighting cyberthreats – tested the vendors on their products’ abilities to detect and respond to real-world attacks in a simulated environment.
LockBit is among the mostly widely used ransomware variants in the world, and Clop was among the most successful last year after bad actors were able to hack into Progress Software’s MOVEit file transfer software. On the other end, North Korea remains among the largest exporters of cybercrime, in large part using it as a way to bypass international sanctions and fund its nuclear and other weapons programs.
North Korean state-sponsored groups are often evolving their tactics, such as developing advanced and multistate malware, and is expanding its targeting of macOS systems.
Trusted Security Tests
The MITRE tests are the most well-known and trusted third-part tests, according to Aviad Hasnis, CTO of Cynet, one of the vendors evaluated.
“All participating vendors face the exact same set of simulated attacks based on the actions of real-world threat groups,” Hasnis told MSSP Alert. “This provides a more realistic evaluation of a solution’s effectiveness than vendor claims or even industry analyst assessments.”
MITRE says the tests are meant to rank vendors or their products. Instead, they give organizations insights and results they can use when evaluating vendors. That said, it gives vendors who perform well a chance to boast about the results.
Cynet, which offers its All-in-One Cybersecurity Platform to MSSPs, MSPs, VARs, and consultants and directly to small and midsize enterprises (SMEs) and public sector organizations, said scored 100% on both detection and protection. Hasnis said Cynet outscoring more well-known vendors is a victory for smaller service providers and organizations.
A Win for the Small Guys
“Cynet’s strong performance signals that enterprise-grade capabilities are no longer reserved for Fortune 500 organizations with big security teams and blank-check budgets,” he said. “Smaller MSPs, MSSPs, and SMEs can now obtain world-class protection at lower cost with less resources required to operate and maintain the solution.”
Organizations, as their IT environments expand, tend to add more cybersecurity tools, creating a collection of standalone solutions that drive costs, complexity, and cracks in defenses.
“This dynamic is especially acute for small-to-medium enterprises and managed service providers, which lack the personnel and resources to constantly implement, integrate, and manage tools, instead of focusing on revenue-generating priorities,” Hasnis said.
Others also pointed out their wins. Sophos’ extended detection and response (XDR) solution scored 100% in detecting, analyzing, and describing the activity of the Clop and LockBit ransomware adversary and 95% in the analysis of the North Korean macOS attack.
“Spoiler alert! Sophos has once again achieved exceptional results in the latest 2024 MITRE ATT&CK Evaluations for Enterprise,” wrote Paul Murray, senior product marketing director at Sophos.
SentinelOne’s Singularity Platform scored 100% in such areas as attack detections and detecting attack techniques across all three operating systems – Windows, Linux, and macOS. There also were no detection delays across the simulated attack scenarios, the vendor wrote.
Disagreement Over Protection
Karthik Selvaraj, partner director for Microsoft’s Defender XDR research team, wrote that the “cyberattack used during the detection test highlights the importance of a unified XDR platform.” However, Selvaraj added that Microsoft’s view was that the protection part of the evaluation did not simulate the real-world cyberthreats organizations face.
“Our conclusion from the MITRE protection test is that it was designed to evade protection mechanisms to the extent that it is unrepresentative of an actual cyberattack, a methodology that Microsoft disagrees with,” he wrote.
At issue was the micro-testing methodology that Selvaraj said was “inconsistent with how cyberattackers typically operate, moving laterally within organizations by gaining access to identities and privileges over time. These broader signals are critical for distinguishing between benign and malicious activities so we can balance protecting organizations from cyberattacks while supporting the broadest set of benign use cases across a massive customer base worldwide.”
