✓ Human-authored analysis; AI used for formatting and proofreading.
We tested a cloud security reasoning engine against intentionally vulnerable AWS environments published by four security research teams. Some of these environments were designed for tool validation and learning purposes. Datadog's Pathfinding Labs asks: "does your tooling catch every vulnerable config?" Others were designed for offensive skill development but serve the same purpose: the attack paths are documented, the correct answers are known, and any tool can be tested against them.
The results found hidden bugs in the tool. Each miss was a precise, actionable product requirement with a known correct answer provided by the security researchers who designed the lab.
This article describes the methodology. It applies to any tool that evaluates configuration snapshots. Tools that read a point-in-time capture of cloud state and evaluate it locally. Tools that require live API access, runtime agents, or network-level monitoring use a different testing approach. The snapshot-based methodology works because the lab infrastructure can be captured at rest, torn down, and re-evaluated offline indefinitely.
The problem: how do you test a security tool?
The standard approach is unit tests and synthetic fixtures. You write a JSON file for an IAM policy with a known misconfiguration, feed it to your tool, and check whether the tool flags it. The test passes. You ship.
The problem is you wrote the test, the tool and the fixture. If your understanding of the vulnerability is incomplete, test and fixture share the same misunderstanding. The test passes because the fixture encodes your misunderstanding.
This is the GiGo problem applied to security tooling. When the person who builds the tool also builds the tests, the tests validate the builder's assumptions, not the tool's correctness.
The solution: someone else already built the fixtures
Security research teams publish intentionally vulnerable AWS environments for exactly this purpose. Each environment deploys real AWS resources with real misconfigurations using Terraform. The attack paths are documented. The correct answers are known. Some platforms like Datadog's Pathfinding Labs explicitly design for blue team tool validation: "does your tooling catch every vulnerable config?"
These are the best possible test suite for security tooling. Because the researchers who designed them are domain experts, they're independent from the tool being tested, and they've already documented the correct detection.
Four teams publish environments we used:
Datadog (Pathfinding Labs) — A modular platform for deploying intentionally vulnerable AWS configurations across multi-account environments. 100+ labs. Designed for two audiences: blue teams (validate CSPM detection, measure coverage gaps) and red teams (practice IAM exploitation, test tooling, build attack chains). The plabs CLI manages deployment, and plabs demo walks the exploit path and captures a flag. This is empirical proof that the attack succeeds. The blue team question: "does your tooling catch every vulnerable config?" is the reason why we used it.
Bishop Fox (IAM Vulnerable) — 31 IAM privilege escalation paths deployed in a single Terraform apply. Each path represents a specific technique documented by security researchers. The environment was explicitly designed for "testing your tools". The README names PMapper, Cloudsplaining, and Pacu as intended consumers.
NCC Group (SadCloud) — 20+ common cloud misconfigurations in one deployment. Unencrypted volumes, open security groups, public buckets, missing logging. Each misconfiguration represents a finding that any CSPM tool should detect.
Rhino Security (CloudGoat) — Multi-step attack scenarios involving multiple AWS services. Each scenario has a walkthrough documenting the exact exploitation path. The scenarios are the most complex test cases, chains spanning Lambda, EC2, S3, and IAM.
The methodology
The process is the same for every lab from every vendor:
1. Deploy the lab (Terraform apply into a sandbox account) 2. Capture a configuration snapshot (AWS CLI + jq) 3. Build an observation from the snapshot (map raw config to your tool's schema) 4. Run your tool against the observation 5. Compare your tool's output to the lab's known answer 6. Every miss is one of three things: A. Wrong field name → fix the observation B. Control exists but has a bug → fix the control C. No control exists → product requirement 7. Fix the miss, rebuild, re-run against the SAME snapshot 8. Verify the fix. Commit with the lab ID for traceability. 9. Tear down the lab (Terraform destroy) The snapshot stays on disk. The lab infrastructure is torn down. Every future iteration runs against the saved snapshot without requiring access to live AWS resources. The snapshot becomes a permanent test fixture, but one that was designed by an independent expert.
The CTF exploit is the oracle. It is proven correct by successful exploitation. The tool must arrive at the same conclusion through a different method (static analysis vs. runtime exploitation). Same answer, two independent methods.
What the labs revealed
We ran a reasoning engine with 2,662 security controls against two vendors.
Datadog Pathfinding Labs: 2 labs, 7 findings
Lab 1 (single-hop privilege escalation): A user holds iam:CreateAccessKey scoped to an admin user. One API call produces long-lived admin credentials.
Tool result: 3 findings. Exact match. The tool detected the escalation on the starting user, flagged the admin target, and identified the cross-cloud admin risk. Every finding matched the lab's documented attack path. Three reasoning engines (Z3, Soufflé, Prolog) independently confirmed the escalation from the exported observation primitives.
Lab 2 (two-hop chain via Lambda): A user can UpdateFunctionCode on a Lambda function whose execution role has iam:CreateAccessKey on an admin user. Two hops:
user → Lambda → role → admin.
Tool result (first run): 3 findings. Hop 2 detected (role → admin). Hop 1 missed (user → Lambda). The starting user had zero findings.
Why it missed: The observation used a field name the tool didn't recognize. The control existed (CTL.IAM.ESCALATE.EDITLAMBDA.001) but the observation said lambda_code_update.present while the control read editlambda.present. A naming mismatch caused the problem.
Fix: Rename the field. Rebuild. Re-run. 4 findings. All hops detected. Reasoning engines confirmed the full two-hop chain through ARN edge traversal. The starting user reaches the admin user through the intermediate Lambda role, traced through specific resource identifiers, not through co-occurrence of alerts.
What this taught us: The control catalog was correct. The field naming convention documentation was insufficient. A human building the observation guessed the field name instead of reading the control YAML. The lab caught it. A unit test written by the same developer wouldn't have. Because the developer would have used the same wrong name in both the test and the fixture.
Bishop Fox IAM Vulnerable: 33 paths, 26 modeled
First run: 24 of 26 findings. 92% detection rate on the first attempt with zero field-name bugs. Because we learned from the Datadog lab 2 mistake and read every control YAML before writing the observation.
The 2 misses (same root cause): Two controls (ATTACHROLEPOLICY.001 and PUTROLEPOLICY.001) gated on identity.kind == role but the Bishop Fox technique uses a USER who holds the permission. The user attaches admin to an assumable role, then assumes it. The escalating principal is a user. The control only matched roles.
Fix: Remove the kind gate. The control should fire on ANY principal that holds the dangerous permission, regardless of whether it's a user or role. Rebuild. Re-run. 26 of 26.
What this taught us: The control logic was correct for the "role modifies its own policy" case but missed the "user modifies a role's policy" case. The Bishop Fox lab surfaced a blind spot the tool's own test suite couldn't catch. Because the tool's tests only covered the role-as-principal case. The researchers designed both cases. The tool only tested one.
Catalog gaps: 5 of 33 Bishop Fox paths had no corresponding control at all. Each is a specific product requirement:
ec2-instance-connect:SendSSHPublicKey SSH into EC2 with higher-privilege role SageMaker PassRole → CreateNotebook PassRole to notebook instance SageMaker PassRole → CreateTrainingJob PassRole to training container SageMaker PassRole → CreateProcessingJob PassRole to processing container SageMaker PassRole → CreateEndpoint PassRole to inference endpoint Each gap is a specific escalation technique documented by Bishop Fox. The technique is known. The correct detection is known. The implementation pattern exists. It is identical to existing PassRole controls for Lambda, EC2, Glue, CloudFormation, and DataPipeline. The only work is authoring the control and the observation fields.
The scorecard tells a story
Vendor Labs Modeled Detected Bugs found Gaps found ────────── ──── ─────── ──────── ────────── ────────── Datadog 2 7 7 2 0 Bishop Fox 1 26 26 2 5 ────────── ──── ─────── ──────── ────────── ────────── Total 3 33 33 4 5 4 bugs found and fixed. 5 catalog gaps identified as product requirements. Every bug had a known correct answer. Every gap had a specific technique with a documented exploitation path. None of these would have been found by the tool's own unit test suite. Because the tool's tests encode the builder's understanding, and the misses were in the gaps of that understanding.
The researchers already prioritized the techniques. Their ordering reflects what they see in real-world attacks. The most important escalation paths are the ones they put first in their getting-started guides. That ordering is the product roadmap. It is derived from expert assessment.
Each vendor finds different bugs
Different vendors test different dimensions. The bugs don't overlap:
Datadog: Field naming conventions Chain linkage across services Observation schema gaps Bishop Fox: Asset type restrictions Control catalog coverage breadth Missing control variants for specific AWS services NCC Group: Per-resource detection gaps (CSPM layer) Service coverage breadth Rhino Security: Compound detection depth Cross-service reasoning One vendor isn't enough. The Datadog labs didn't reveal the asset type bug, only Bishop Fox's technique (user-holds-role-permission) triggers it. Bishop Fox didn't reveal the field naming bug, only Datadog's multi-hop chain requires building observations manually where the naming mismatch appears. Each vendor tests a different surface of the detection problem. The combination provides comprehensive coverage validation.
Why exploit demonstrations are the strongest oracles
Pathfinding Labs includes a plabs demo command for each lab that walks the exploit path step by step and captures a flag. The flag is proof. It is empirical proof that the attack path works against real AWS resources.
plabs demo proves the path EXISTS: Security team runs plabs demo <lab-name> → Walks the exploit chain step by step → Captures the flag → The path is proven exploitable Tool must detect the same path from configuration alone: Tool reads the snapshot → Analyzes the relationships between resources → Reports the chain as a finding plabs demo captured the flag + tool missed the path = BUG plabs demo captured the flag + tool found the path = CORRECT A captured flag is real. The exploit ran via plabs demo. The flag was captured. The path works. If the tool doesn't detect it, the tool is wrong without any ambiguity.
Tool validation is a designed use case
Pathfinding Labs documents two audiences: red teams who practice exploitation, and blue teams who validate detection tooling. The blue team workflow is: deploy the lab, run your CSPM tool, measure what it catches, identify coverage gaps.
This article describes that by using the labs as Datadog intended for blue team validation. Bishop Fox's IAM Vulnerable similarly states its purpose as testing tools like PMapper, Cloudsplaining, and Pacu. The researchers built these environments for tool builders. The tool-testing is the documented purpose.
What the tool builder adds to the relationship:
Validation that the labs work. If the Terraform doesn't deploy correctly, the tool builder reports it. The lab infrastructure is being tested too.
Evidence that the labs matter. "We tested against Bishop Fox IAM Vulnerable and found 5 gaps in our catalog" validates the researchers' work as the industry standard for privilege escalation coverage.
Feedback on new techniques. When a tool has a control that no vendor lab covers, that's a signal: "here's a technique your lab could add." The feedback loop benefits both sides.
How to start this week
This methodology applies to tools that evaluate configuration snapshots offline. Checkov, OPA, Prowler, ScoutSuite, Stave, or any tool that reads JSON/YAML configuration and produces findings without needing live API credentials during evaluation.
If your tool requires live API access or runtime monitoring, the lab infrastructure must stay deployed during evaluation. The snapshot-based approach below won't apply but the oracle principle (known-input, known-output, independent source) still applies.
Day 1: Deploy Bishop Fox IAM Vulnerable (free, IAM-only, 31 paths in one terraform apply). Capture the snapshot. Run your tool. Build the scorecard.
Day 2-3: Fix the misses. Each miss has a known answer from Bishop Fox. Rebuild. Re-run against the same snapshot. The snapshot doesn't change. Only the tool changes.
Day 4: Deploy one Datadog Pathfinding Lab (free, IAM-only). Run the CTF to establish the oracle. Snapshot. Run your tool. Compare to the CTF's known path.
Day 5: Update the scorecard. Commit every fix with the lab ID. You now have a detection coverage map validated against two independent expert sources. This is something no unit test suite can provide.
The labs are free. The sandbox account costs nothing (IAM-only resources). The snapshot-based methodology is a one-time investment that produces a permanent, growing test suite maintained by security researchers. The snapshots persist on disk after the infrastructure is torn down. Every future test runs offline against the captured state. The oracle library grows every time the vendors publish a new lab. Capture the snapshot once, re-evaluate indefinitely.
Stave uses this methodology across four vendor oracle sources. Every bug fix is traceable to the specific vendor lab that revealed it. The scorecards, observations, and results are committed to the repo. The test suite grows with every vendor lab we run and every miss becomes a shipped control with a known correct answer.