ATT&CK EVALUATIONS:
Using Evaluations
How To Use & Extract Value from MITRE Engenuity ATT&CK® Evaluations
Knowing how to review ATT&CK Evaluations best empowers you to pick the vendor that is most relevant to your organization’s needs and helps you understand your tool. Many people stop their analyses of ATT&CK Evaluations at the summary, but it’s important to go beyond because:
- Each section informs the other for a meaningful analysis.
- The sum of what you discover in the details is greater than the whole found in single-number results.
Remember, we don’t compare or rate providers or tools. We only evaluate individual tools.
Tips To Best Understand & Leverage ATT&CK Evaluations
ATT&CK Evaluations are a starting point.
We use an open-book and minimally sized environment to understand baseline capabilities of solutions. Operationalization of these solutions is important to consider in the context of your organization, including false positive generation.
There are no winners.
The goal of ATT&CK Evaluations is to show the different capabilities of each provider.
Not all techniques are created equal.
A technique detection for credential dumping may not have the same value as a technique for process discovery due to the severity of the actions. The category gives you a general idea, but you should dive into the details to understand the technique and detection.
Look at the UI/UX view for more than aesthetics.
Consider how the UI/UX would work, not just the look and feel of it. Are you seeing the information you’d need in real-time?
Determine how a provider’s tool presents itself to your analysts.
Thumb through provider screenshots to see if you’re being presented the data you need and review a tool’s overall usability. Is it easy to find the data you want to see in real-time?
See if a tool detects and prioritizes known threats to your organization.
Understanding a tool’s detection and prioritization capabilities will help you know if you’ll get an alert at the right time and in a way that allows you to trigger effective deterrence.
Understand the data sources: Are they effective?
For example, from what data source was a detection triggered, and how does that trigger fit into your cybersecurity needs and tactics?
KEY PARTS OF AN ENTERPRISE ATT&CK EVALUATION
-
Evaluation Overview Page
Access an Enterprise Evaluation Overview
Each Enterprise evaluation has available information specific to those evaluations. The overview page for each round provides high-level information and includes:
- Links to the results
- A list of participants
- A description of the adversary
- Emulation notes
- The technique scope
- Environment notes
- Detection categories
- Related resources about the evaluation
-
Detection Summary Screenshots
A detection summary shows a collection of screenshots from the provider. This allows users to quickly review the UX/UI of the tool.
-
Detailed Results & Screenshots
Within Evaluations results for every provider, there are specifics on each scenario and details for every step and sub-step of the emulation plan including:
- Tactic, technique, and sub-technique information
- Detection categories
- Detection criteria and data sources
- Screen shots of detections within the provider’s environment
-
Protection Results
These are linear tests with a defined start and end. They are opt-in and not every result includes them.
-
Participant Configuration
This includes provider product versions and descriptions used in the evaluation.
-
JSON (Java Script Object Notation) File
The JSON file can be downloaded to dive more deeply into results and build internal analytics to further parse data.
Managed Services 2023 Call for Participation
If you’re a managed services provider looking to help current and prospective clients understand more about how your service addresses known adversary behavior, Evaluations are a phenomenal platform to showcase your service and highlight your competitive advantage to your audience. We recommend that organizations explore participation if they are looking to:
- Earn the community’s trust. Evaluation rounds are transparent and openly published.
- Improve your capabilities and learn to better defend against the adversary.
Evaluations by Industry
The ATT&CK Evaluations program continues to develop new methodologies, and open new rounds of evaluations. Currently, there are four types of ongoing ATT&CK Evaluations available:
Enterprise
ATT&CK® Evaluation for Enterprise empowers end-users to make more informed decisions on endpoint detection capabilities by articulating how each vendor can protect against or detect adversary behavior.
Managed Services
ATT&CK® Evaluations for Managed Services provides transparent and impartial insights into how managed security service providers (MSSPs) and managed detection and response (MDR) capabilities provide context of adversary behavior.
Industrial Control Systems (ICS)
MITRE Engenuity ATT&CK® Evaluations for Industrial Control Systems (ICS) clarify anomaly and threat detection capabilities of industrial control systems security solutions.
Trials
ATT&CK® Evaluation Trials allow more capabilities that do not fit in the aforementioned categories to be evaluated. The first Trial is for Deceptions.
MOre resources
Get Involved
Apply to be part of the ATT&CK Evaluations Community Advisory Board
Our commitment to innovation and consistency requires deeper interaction in a more structured format with the community we are looking to benefit. Our goal is to learn how we can make the data we present more accessible to the community at large. As an end user of ATT&CK Evaluations and a member of the infosec community, your insights are valuable to us.
Stay informed.
Sign up for news about MITRE Engenuity and the critical technical challenges facing our nation and world.