Security Week

AMTSO Releases Sandbox Evaluation Framework

Eduard Kovacs March 26, 2025 2 min read

AMTSO, the cybersecurity industry’s testing standards community, on Wednesday announced the creation of a sandbox evaluation framework whose goal is to standardize the testing of sandbox-based malware analysis solutions.

Sandbox systems are increasingly important in the analysis of malware and other potential threats, but it can be challenging to determine which solution is the best for a user’s specific requirements.

AMTSO’s Sandbox Evaluation Framework aims to address this by providing a list of criteria and a scoring system to help researchers, vendors and other members of the cybersecurity industry with evaluating and comparing sandboxes.

For instance, inline protection sandboxes have a very low latency and are good for real-time protection, which makes them ideal for products such as email gateways and web application firewalls. However, their analysis capabilities in terms of depth are limited.

On the other hand, full attack chain analysis sandboxes are much slower, but their depth capabilities are very high, enabling the analysis of sophisticated threats.

AMTSO’s Sandbox Evaluation Framework looks at a sandbox’s detection capability, anti-evasion technology, analysis depth, speed and scale, deployment, reporting and threat intelligence, and automation and integration.

“Each of these indicators addresses a critical aspect of sandbox efficacy, allowing organizations to make informed decisions about which solution best fits their security needs,” the framework’s developers said.

“For example, an organization focusing on a prevention use case may favor detection capability, speed, and scalability. An email security gateway vendor that needs to process a massive amount of files may favor detection capability, compute cost, and ease of deployment/maintenance, or a research lab might be interested in deep-diving memory dumps and dissecting a file from an incident response perspective,” they explained.

The documentation shared by AMTSO explains how scores can be assigned — for example, 0 is given if a feature is not available, 3 for limited support, and 10 for exceptional capability. It also explains the process of assigning weights depending on the importance of each performance indicator.

Advertisement. Scroll to continue reading.

Once scores and weights have been assigned, the user can calculate a total score and a weighted score that indicates which sandbox solution is the best for their needs.