How I built an AWS forensic response pipeline with Step Functions, Python, Terraform, cross-account KMS and a human approval gate. Plus the Lambda timeout that reminded me a snapshot copy is not a function call.

The 3 AM Problem

It's 3:07 AM. GuardDuty flags an EC2 instance talking to a known malicious IP.

Someone gets paged. They open Slack, check the finding, and search Confluence for the incident response runbook. The last edit was eight months ago.

Step 4 says "Snapshot the volumes before remediation."

Simple enough, until you're signed into the wrong AWS account, using a role that cannot read the encryption key, wondering whether the snapshot should happen before or after isolation.

Then someone says the sentence every security engineer loves to hear at 3 AM:

"Can we just terminate the instance?"

No. Not yet.

A runbook can tell people the correct order. It cannot stop an exhausted engineer from doing things in the wrong order.

That bothered me enough to try a different approach. Instead of writing another page of instructions, I encoded the response as a state machine.

An incident could advance only after the previous step had produced the evidence required to continue.

This is the design, the failures, and the changes I'd make now. A few implementation details are generalized because the workflow was built in an organizational environment. The code excerpts are representative, not a complete drop-in deployment.

Why I Didn't Just Write a Python Script

I considered it. One Python file, a bunch of boto3 calls, a few try/except blocks, and done.

Except the workflow needed to:

  • isolate an instance without casually destroying evidence;
  • capture and track every relevant EBS snapshot;
  • move copies into an account with separate security ownership;
  • wait for snapshot operations that might take much longer than a Lambda invocation;
  • handle failures without blindly repeating expensive operations;
  • pause for an actual person to authorize remediation;
  • explain, later, who approved what and which artifacts were preserved.

A Python script can perform those actions. Keeping its state correct across crashes, retries, approvals and multi-day investigations is the hard part.

I chose AWS Step Functions Standard Workflows because they support durable execution history, per-state retry policies and callback tasks. Express Workflows do not support the waitForTaskToken pattern used for the approval gate.

There is still code. I just stopped making the code responsible for remembering the entire incident.

The Architecture

 GuardDuty / Security Hub | EventBridge | v +------------------------------------+ | RESPONSE / WORKLOAD ACCOUNT | | | | Step Functions (Standard) | | | | Validate finding + deduplicate | | | | | Isolate with approved controls | | | | | Capture EBS snapshots | | | | | Share snapshot access securely | +---------------+--------------------+ | Assume narrow forensic role | v +------------------------------------+ | FORENSIC ACCOUNT | | | | Copy snapshot with forensic CMK | | | | | Wait / poll / verify completion | | | | | Record evidence metadata | +---------------+--------------------+ | v HUMAN APPROVAL / \ approve reject / timeout | | v v Remediate Remain isolated + escalate 

I intentionally designed two different kinds of boundaries.

The workflow boundary controls order. Remediation cannot begin through this workflow until the forensic copy is verified and approval is received.

The account boundary controls ownership. Once the evidence is copied into the forensic account and encrypted with a forensic-controlled customer managed KMS key, the workload account does not need access to decrypt that copy.

Important distinction: a state machine does not stop someone with unrelated administrator access from terminating an instance outside the workflow. Preventing that requires IAM guardrails, SCPs where appropriate, break-glass controls and a tested incident process. The machine makes the automated path safe by default.

The stack

Concern What I used Why
Orchestration Step Functions Standard Durable execution, history and approval callbacks
Actions Python Lambda dispatcher with 16 handlers Shared code and one deployment artifact
Infrastructure Terraform Reviewable policies, roles, keys and workflow definitions
Evidence boundary Separate forensic AWS account Separation from the compromised workload environment
Encryption Customer managed KMS key in the forensic account Control who can use the copied evidence
Approvals waitForTaskToken plus a protected API A person authorizes destructive action
Recovery state DynamoDB Track incident actions, copy attempts and approvals

The single dispatcher Lambda was a pragmatic decision, not a security ideal. I'll come back to that.

Failure 1: I Asked Lambda to Wait for an EBS Snapshot

The first version looked reasonable:

  1. Call CopySnapshot.
  2. Wait until the snapshot becomes completed.
  3. Return the snapshot ID.

It worked on a small test.

Then I tried a larger snapshot. The Lambda timed out while AWS was still doing exactly what I had asked it to do.

I had misunderstood the boundary between starting a job and finishing a job.

CopySnapshot starts an asynchronous operation and returns a new snapshot ID. The snapshot can still be pending long after the API call succeeds.

A Lambda invocation has a maximum runtime of 15 minutes. A snapshot copy does not promise to finish within that window. The default boto3 snapshot_completed waiter also has a finite polling budget, and that budget is not a guarantee of completion.

There was another surprise. Changing the KMS key when copying an encrypted snapshot makes the copy a full copy rather than an incremental one. That affects time and cost, especially when you are preserving evidence across accounts.

My first attempt had turned a long-running storage operation into a blocked function call.

The fix: start in Lambda, wait in Step Functions

The Lambda now starts the copy and immediately returns the ID. Here is the important part, with role/session setup omitted:

@action("start_snapshot_copy") def start_snapshot_copy(ctx): ec2 = ctx.forensic_ec2() result = ec2.copy_snapshot( SourceRegion=ctx.source_region, SourceSnapshotId=ctx.source_snapshot_id, Encrypted=True, KmsKeyId=ctx.forensic_kms_key_arn, TagSpecifications=[{ "ResourceType": "snapshot", "Tags": [ {"Key": "IncidentId", "Value": ctx.incident_id}, {"Key": "SourceSnapshotId", "Value": ctx.source_snapshot_id}, {"Key": "Purpose", "Value": "forensic-evidence"}, ], }], ) return {"copy_snapshot_id": result["SnapshotId"]} @action("check_snapshot_copy") def check_snapshot_copy(ctx): snapshot = ctx.forensic_ec2().describe_snapshots( SnapshotIds=[ctx.copy_snapshot_id] )["Snapshots"][0] return { "state": snapshot["State"], "progress": snapshot.get("Progress"), } 

The state machine owns the loop:

{ "WaitForCopy": { "Type": "Wait", "Seconds": 60, "Next": "CheckCopy" }, "CheckCopy": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": { "FunctionName": "${dispatcher_arn}", "Payload": { "action": "check_snapshot_copy", "copy_snapshot_id.$": "$.copy.copy_snapshot_id", "incident.$": "$.incident" } }, "ResultSelector": { "state.$": "$.Payload.state" }, "ResultPath": "$.copy_status", "Next": "CopyFinished" }, "CopyFinished": { "Type": "Choice", "Choices": [ { "Variable": "$.copy_status.state", "StringEquals": "completed", "Next": "VerifyEvidence" }, { "Variable": "$.copy_status.state", "StringEquals": "error", "Next": "CopyFailed" } ], "Default": "WaitForCopy" } } 

That is an excerpt from the state machine, not a complete ASL definition. In the real workflow, a deadline and retry budget prevent a copy from sitting in the poll loop indefinitely. Other exceptional states also reach an explicit failure or escalation path.

A Wait state does not burn Lambda compute while it waits. It does consume Step Functions state transitions, so "free waiting" isn't literally free.

Lesson: If a job can outlive a function invocation, let the function start it and let the orchestrator track it.

The trap behind the trap: retries can create duplicate evidence

My initial solution used snapshot tags as an idempotency check. Search for IncidentId, and if a copy exists, return it.

Sounds good. It isn't a lock.

Two invocations can both query tags, both see nothing, and both call CopySnapshot. Worse, an API call can succeed while the Lambda crashes before saving the returned ID. A retry has no reliable way to know whether the first copy started.

So I added an incident-scoped operation record in DynamoDB.

operation_id = incident_id + source_snapshot_id + destination_region NEW | | conditional claim (only one worker wins) v STARTING | | CopySnapshot returns snapshot ID v COPYING | | DescribeSnapshots says completed v VERIFIED 

If another invocation sees STARTING, it does not automatically issue another copy. It waits for reconciliation, using the operation record and the tagged snapshot inventory to investigate an ambiguous result.

This is an important limit: CopySnapshot does not offer a general client idempotency token in its request. A DynamoDB conditional write prevents competing workers from starting the same logical operation, but an uncertain network response still requires careful reconciliation. I would rather delay a copy than silently create two pieces of evidence with confusing provenance.

Failure 2: Cross-Account KMS Has More Than One Lock

I expected the hardest part to be copying the snapshot.

It was permissions.

One role could describe the source snapshot but could not copy it. Another role could call the EC2 API but received a KMS access error. The error message did not come with a diagram showing which policy I had missed.

The mental model that finally helped me was this:

Source snapshot access | +-- source snapshot sharing permissions | +-- source / associated KMS key policy | +-- forensic copier IAM permissions | +-- forensic destination KMS key permissions | v Encrypted copy in forensic account 

Both the cross-account key policy and the caller's IAM policy matter. Granting one without the other is not enough.

A deliberately simplified Terraform fragment for a source-side customer managed key looks like this:

data "aws_iam_policy_document" "forensic_key_access" { statement { sid = "AllowForensicCopier" effect = "Allow" principals { type = "AWS" identifiers = [var.forensic_copier_role_arn] } actions = [ "kms:DescribeKey", "kms:Decrypt", "kms:ReEncrypt*", "kms:GenerateDataKey*" ] resources = ["*"] } statement { sid = "AllowEbsGrantForForensicCopier" effect = "Allow" principals { type = "AWS" identifiers = [var.forensic_copier_role_arn] } actions = ["kms:CreateGrant"] resources = ["*"] condition { test = "Bool" variable = "kms:GrantIsForAWSResource" values = ["true"] } } } 

This is a policy fragment, not a complete key policy. The copier role also needs narrowly scoped IAM permissions, and the forensic destination key needs a policy that allows EBS to encrypt the copy and approved forensic roles to use it. An investigator should not automatically inherit the copier's privileges, and a workload operator should not inherit the investigator's.

A November 2026 AWS change makes this more interesting

There is an important update to the old advice that a snapshot encrypted with aws/ebs must first be copied to a customer managed key before sharing it.

In November 2026, Amazon EBS introduced associated KMS keys for encrypted snapshots. You can associate an additional customer managed key with an encrypted snapshot, including one encrypted under aws/ebs, and grant another account access through that associated key. Associating the key does not re-encrypt the existing snapshot. That avoids an otherwise necessary source-side re-encryption copy in supported workflows.

This doesn't eliminate the destination copy. My forensic design still copies the snapshot into the forensic account and re-encrypts that copy under a key owned by the forensic team. Those are two different security goals:

  • Associated source key: securely makes an existing snapshot available for cross-account copying.
  • Forensic destination key: controls the independent copy after it lands in the forensic account.

Before this capability, the source-side re-encryption copy was often the practical route for aws/ebs snapshots. If you're designing this today, check AWS's associated KMS key documentation instead of copying an old diagram from 2024.

The final checklist I use is straightforward: who owns the snapshot, who may read its data, who can initiate a copy, who owns the destination key, and who can destroy the forensic copy?

Lesson: Don't debug KMS by adding permissions until the error disappears. Identify the exact boundary that is failing.

Failure 3: The Approval Gate Was a Bigger Security Risk Than the Snapshot

Step Functions has a feature I wish more people used correctly: waitForTaskToken.

It can pause a Standard workflow while an external system makes a decision. No Lambda needs to stay alive while someone reviews the evidence.

My first design had a straightforward approval step:

{ "RequestApproval": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken", "TimeoutSeconds": 3600, "Parameters": { "FunctionName": "${dispatcher_arn}", "Payload": { "action": "request_approval", "task_token.$": "$$.Task.Token", "incident.$": "$.incident" } }, "ResultPath": "$.approval", "Catch": [ { "ErrorEquals": ["ApprovalRejected", "States.Timeout"], "ResultPath": "$.approval_error", "Next": "KeepIsolatedAndEscalate" } ], "Next": "Remediate" } } 

Without a timeout, a callback task can remain waiting until the workflow's execution limit. A quarantined machine could sit there long after everyone has forgotten why it was isolated.

The first rule became: no response is not approval.

A rejection or timeout keeps the instance isolated and escalates to an investigator. The isolation remains under active incident ownership, rather than turning into an abandoned quarantine.

Then I noticed something worse in the usual online examples.

A task token in an approval link is basically a password

Lots of tutorials put the callback token inside an email URL:

https://example.com/approve?token=<STEP_FUNCTIONS_TASK_TOKEN> 

Anyone who obtains that token and has a path to the callback API can potentially complete the task. Email forwarding, link previews, logs and browser history become part of your authorization model. That's not an authorization model I want.

I changed the design:

  1. The backend stores the task token in an encrypted, access-controlled DynamoDB record.
  2. The notification contains only a random, opaque approval_id.
  3. The user signs in through SSO and uses an authenticated POST, not an approval GET link.
  4. The backend checks the approver's role, incident scope, expiry and separation of duties.
  5. A conditional DynamoDB update claims the decision so two people cannot independently process the same pending approval.
  6. A callback worker sends SendTaskSuccess or SendTaskFailure and records the delivery result.

I also disallow self-approval. An operator can request destructive remediation, but a different authorized person must approve it.

That extra callback worker is not overengineering. A DynamoDB update can succeed and the subsequent Step Functions callback can fail. Atomicity inside DynamoDB does not make DynamoDB and Step Functions one atomic transaction.

My approval record distinguishes PENDING, PROCESSING and DELIVERED. Failures are retried or reconciled without quietly treating a locally recorded decision as an accepted callback. Explicit expiry is checked at request time; DynamoDB TTL is cleanup, not an exact approval timeout mechanism.

Finally, the token stays out of application logs. Step Functions execution history and Lambda invocation access are also restricted. Disabling full execution payload logging is useful, but it does not make task-token-bearing execution metadata public-safe.

logging_configuration { log_destination = "${aws_cloudwatch_log_group.sfn.arn}:*" level = "ERROR" include_execution_data = false } 

Lesson: Human approval needs a deadline, authenticated identity, separation of duties and a safe default. Otherwise it is just another button.

Failure 4: I Let Step Functions Retry an Action It Shouldn't Retry

Step Functions makes retrying tasks almost too easy.

My first policy treated transient throttling and permanent authorization errors roughly the same. That was a mistake.

A missing permission does not become a valid permission after sleeping for five seconds. Meanwhile, retrying a state that starts an external operation can duplicate the operation if that state is not designed for retries.

I ended up separating failure types at the Lambda boundary:

class TransientError(Exception): pass class PermanentError(Exception): pass def raise_aws_error(exc): code = exc.response.get("Error", {}).get("Code", "Unknown") if code in { "Throttling", "ThrottlingException", "RequestLimitExceeded", }: raise TransientError(code) from exc raise PermanentError(code) from exc 

And I gave retrying a clear opinion:

"Retry": [ { "ErrorEquals": [ "TransientError", "Lambda.ServiceException", "Lambda.TooManyRequestsException" ], "IntervalSeconds": 5, "MaxAttempts": 4, "BackoffRate": 2.0 } ] 

An authorization error goes to escalation. A transient throttling error can be retried. An ambiguous snapshot-copy result goes to the reconciliation path, not straight back into CopySnapshot.

I also kept the dispatcher intentionally dull:

HANDLERS = {} def action(name): def register(fn): HANDLERS[name] = fn return fn return register def lambda_handler(event, context): handler = HANDLERS.get(event.get("action")) if handler is None: raise PermanentError("Unsupported action") return handler(Context(event)) 

A boring dispatcher is a good dispatcher. It should route, validate and fail loudly. It should not silently guess what to do with an unfamiliar action.

Lesson: Retries are a distributed-systems design decision, not a checkbox.

The Security Problem I Would Fix If I Built It Again

I started with one Lambda and 16 handlers because deployment was simpler.

There was one artifact to package. Shared clients, decorators, typed errors and tests all lived in one place. For a first implementation, that was useful.

The downside is the IAM role. A single Lambda execution role tends to accumulate the union of permissions required by all 16 handlers.

That is convenient for a developer and attractive to an attacker.

I would split it into a thin dispatcher and capability-specific roles, or separate Lambdas for high-risk actions:

Step Functions | +--> Validate finding (read-only role) | +--> Isolate instance (containment role) | +--> Snapshot volumes (snapshot role) | +--> Copy forensic evidence (cross-account copier role) | +--> Approval callback (callback-only role) | +--> Remediate (restricted remediation role) 

The dispatcher should not have ec2:TerminateInstances just because one of its handlers might eventually need to terminate an instance.

For cross-account trust, I prefer an exact allowed principal and, for accounts in the same AWS Organization, an aws:PrincipalOrgID condition where it fits the trust model. ExternalId can be useful for a third-party or confused-deputy scenario, but it is not a substitute for scoping the principal correctly.

The bigger design choice is how much power the incident-response automation receives before an incident. That deserves a security review of its own.

The Forensics Part People Usually Skip

A successful EBS snapshot is not automatically a complete forensic investigation.

A snapshot captures disk state. It does not capture RAM. It might be crash-consistent rather than application-consistent. The environment may change when you isolate it, and some live evidence may disappear.

For an incident that justifies volatile-memory capture, the response plan needs a vetted collection path and a clear decision about what can safely happen before isolation. That decision is threat-dependent. Stopping active exfiltration and preserving volatile evidence can conflict, and no universal step order resolves every case.

I also want evidence metadata that can survive a later investigation:

Field Why it matters
Incident ID and finding ID Connects the evidence to the detection
Source account, instance and volume IDs Identifies where it came from
Source and destination snapshot IDs Connects the original to the preserved copy
KMS key ARNs and copy region Records the encryption boundary
Start, completion and verification timestamps Reconstructs the timeline
Workflow execution ARN Shows which automation acted
Approval identity and decision Establishes authorization
Copy verification result and retention controls Shows the artifact was preserved

Do not confuse recording a snapshot ID with computing a cryptographic hash of all disk contents. If byte-level hashing is part of your chain-of-custody requirements, collect and verify that evidence separately with a repeatable process.

I would also assess EBS Snapshot Lock or equivalent retention controls for evidence that must resist accidental or malicious deletion, along with independent audit-log retention. A separate AWS account is a strong boundary, but it is not magic if the forensic account itself is compromised.

What I'd Actually Measure

I won't invent a benchmark because a polished chart with fake numbers is worse than no chart.

The useful metrics are:

  • Alert to verified containment time, measured at the network boundary, not just when an API returned success.
  • Time to start capturing snapshots and time to verify the forensic copies.
  • Percentage of incidents with complete evidence metadata and an auditable approval decision.
  • Failed or duplicate copy attempts per incident.
  • Approval latency, timeout rate and incidents left in quarantine without active ownership.

I'd also run a scheduled tabletop or safe test incident. IAM policies drift, keys rotate, notification routes change, and a workflow that only worked during its first demo is not an incident-response capability.

Four Lessons That Apply to More Than Incident Response

1. Long-running work belongs in the orchestrator.

This applies to EBS snapshots, data migrations, ML jobs and external payment processors. Start the work, record its identity and monitor it separately.

2. Encode the order, not just the steps.

"Evidence before destructive remediation" is not a paragraph in a document anymore. It's a precondition the workflow enforces.

3. Treat callbacks like security boundaries.

A task token is a credential. Approvals need real identity, expiry, auditability and a safe failure path.

4. Assume every external operation can return an ambiguous result.

A timeout doesn't always mean nothing happened. Plan for reconciliation before you press Retry.

TL;DR

I turned an AWS incident-response runbook into a Step Functions Standard workflow, with Python Lambda handlers, Terraform-managed infrastructure, cross-account forensic snapshot copies and a human approval gate.

The mistakes were the valuable part:

  • I waited for a snapshot inside Lambda, until Lambda timed out.
  • I underestimated how many permission boundaries cross-account KMS introduces.
  • I nearly turned an approval task token into a bearer credential in a URL.
  • I treated retries as harmless before thinking about duplicate forensic operations.

And one recent AWS change matters: associated KMS keys can now simplify sharing encrypted snapshots, including snapshots originally encrypted with aws/ebs. That does not remove the need to control the destination evidence and its encryption key.

An incident response runbook is useful documentation. A state machine is how I stopped hoping everyone would follow it in the right order.

How does your team handle incident response today? SOAR, Step Functions, a collection of scripts, or a Confluence page someone has to find at 3 AM?

I'm especially interested in the failures nobody puts on architecture diagrams.

References: AWS Step Functions callback patterns, EBS snapshot copy behavior, EBS associated KMS keys, Step Functions execution history and logging.