If you run AWS at any real scale, you eventually accumulate thousands of stale EBS snapshots — leftovers from decommissioned volumes, forgotten AMIs, one-off backups nobody ever cleaned up. Cloud Custodian (c7n) is a popular way to find and delete them on a schedule. This post walks through four pitfalls we hit while building a "safe" cleanup pipeline, and the general lessons underneath them — useful for anyone running Custodian, or any policy-as-code cleanup tool, against a large fleet. The setup The pattern is simple and common: Mark snapshots older than N days for deletion (tag them, don't touch them yet). Some days later, copy each marked snapshot as a safety net, then delete the original. After a retention window, age off the safety copies too. Custodian supports all of this natively with a mark-for-op / marked-for-op tag pair and copy / delete actions. Stripped to its essentials, the three stages look like this: # Stage 1 -- mark eligible snapshots, touch nothing else policies: - name: snapshot-mark-eligible resource: aws.ebs-snapshot filters: - type: age days: 90 op: greater-than - "tag:cleanup-exclude": absent actions: - type: mark-for-op tag: cleanup_marked op: delete days: 0 # Stage 2 -- copy-then-delete, gated by the marked-for-op tag policies: - name: snapshot-copy-then-delete resource: aws.ebs-snapshot filters: - type: marked-for-op tag: cleanup_marked op: delete - "tag:cleanup-exclude": absent actions: - type: copy target_region: us-east-1 target_key: alias/aws/ebs encrypted: true tags: safety-copy: "true" - type: delete # Stage 3 -- age off the safety copies after a retention window policies: - name: snapshot-copy-age-off resource: aws.ebs-snapshot filters: - "tag:safety-copy": present - type: age days: 14 op: greater-than actions: - type: delete Run each stage with c7n-org, scoped to every account in your fleet: # Always dry-run first -- writes matched resources to output/ without acting c7n-org run -c accounts.yml -u policies/snapshot-copy-then-delete.yml \ -r us-east-1 -s output --dryrun --cache-period 0 # Scope to a single account while validating a change c7n-org run -c accounts.yml -u policies/snapshot-copy-then-delete.yml \ -r us-east-1 -a my-account --dryrun --cache-period 0 # Execute for real once the dry-run output looks right c7n-org run -c accounts.yml -u policies/snapshot-copy-then-delete.yml \ -r us-east-1 --cache-period 0 On paper, it's a clean, auditable, reversible three-stage workflow. In practice, four assumptions quietly broke it. Worth being precise about two different things that both look like "pipeline" here, because they fail differently: The three stages (mark → copy+delete → age-off) really are a pipeline in the classic sense — each one's output (a tag, with a date on it) is the next one's input, separated by real elapsed time. Pitfalls here look like "the wrong things get picked up at the wrong stage." How a single stage actually executes is not a pipeline at all — it's a fan-out worker: loop over accounts independently (no shared state, each one succeeds or safely retries on its own), and within an account, loop over batches of snapshots capped by a concurrency limit. Pitfalls here look like "something got throttled" or "a batch half-finished." Keep those two mental models separate and the rest of this post is easier to place: pitfalls #1, #2, and #4 are stage-pipeline pitfalls; pitfall #3 is a fan-out-worker pitfall. What actually runs this The mental model above — an isolated worker per account — makes "just run it in Lambda" a tempting read. In practice, a bare Lambda struggles with this specific workload, for three reasons: The 15-minute execution wall. Snapshot copies — especially cross-account/region ones, or anything re-encrypting with a new KMS key — routinely take longer than 15 minutes for volumes in the 100+ GB range. A Lambda running the copy-then-poll-then-delete sequence in one invocation will hard-timeout mid-batch. Wasted polling cost. wait_for_copies below is a time.sleep() loop. Inside Lambda you're billed for vCPU/memory for the whole time the execution context sits there waiting on an AWS storage backend to finish work Lambda has no part in. Alignment with Stages 1 and 3. Those two already run via c7n-org across the fleet, which means a persistent workspace, established cross-account IAM role chaining, and a CLI toolchain are already in place. A scheduled container (ECS Fargate or AWS Batch) or a CI/CD runner (GitHub Actions, GitLab CI) reuses all of that directly; a Lambda would need its own separate deployment and permissions story. If you do want this serverless, the fix isn't a bigger Lambda timeout — there isn't one — it's decomposing the work: AWS Step Functions orchestrating short InitiateCopy → native Wait state → CheckStatus → DeleteOriginal Lambdas, so no single invocation is ever blocked on a copy that runs for hours. Absent that decomposition, a scheduled container or CI job is the simpler, better-fitting choice — and it's what the rest of this post assumes. One more note before the pitfalls: the fixes below replace Stage 2 — copy-then-delete is the unsafe part, so that's the piece that moves from a declarative Custodian policy to a script you control directly via boto3. Stages 1 (mark) and 3 (age-off) are simple tag filters with no ordering or concurrency hazard, so they're left running as Custodian policies exactly as shown above; only the middle stage changes shape. Pitfall #1: "copy first, then delete" doesn't mean what you think A policy like this looks safe: actions: - type: copy target_region: us-east-1 target_key: alias/aws/ebs encrypted: true - type: delete Actions run in order — copy, then delete — against the same matched resource set, in a single policy pass. The instinct is to read that as "the copy will exist before the original is removed." It doesn't say that. CopySnapshot is an asynchronous AWS API call. It returns immediately with a new snapshot ID in pending state; the actual data copy can take anywhere from seconds to hours depending on size. Custodian's copy action, for same-region copies, does not wait for that to finish — it fires the API call and moves straight to tagging the result, and the policy then moves straight to the delete action. There is no dependency enforced between "the copy exists" and "the copy is actually usable." The failure mode: if a copy later fails (hits a transient error, a KMS permission hiccup, a service-side throttle) after the original has already been deleted, you're left with a snapshot in error state and no way to recover the data it was supposed to be safety-netting. This isn't a theoretical edge case — AWS's own CopySnapshot API reference confirms it directly: a copy can return successfully and still fail asynchronously afterward, moving to error state with no dependency on anything that happened before it. It doesn't take a high failure rate to matter here: on data you can't regenerate, even a rare miss is a real loss, not a rounding error — and early on, running this by hand, that's exactly what forced the move away from manual batches: babysitting every run closely enough to catch a late copy failure before delete ran stopped being sustainable well before the fleet did. The fix: don't let delete run in the same breath as copy. Split the two into separate steps, and gate the delete on the copy reaching a terminal, successful state: Trigger the copy (via the cloud API directly, or via your policy engine). Poll the copy's status until it's completed or error (with a timeout). Only delete the original for snapshots whose copy reached completed. Anything that errored or timed out gets left alone and retried next run. This also has a nice side effect: because you're now calling the copy API yourself instead of relying on the policy engine's internal bookkeeping, you get an exact mapping between "this original" and "this copy" — instead of trying to reconstruct it after the fact from a Description field, which (as pitfall #2 below shows) is not reliable. def copy_batch(ec2, region, source_ids, target_key, encrypted=True): """Copy each source snapshot directly via boto3 so the source-id -> copy-id mapping is exact, not guessed from a Description field.""" mapping = {} for source_id in source_ids: # Build kwargs conditionally: botocore rejects KmsKeyId=None outright # (it's a string-typed field), so only include it when target_key is # actually set. The API also requires Encrypted=True whenever # KmsKeyId is supplied -- enforce that here so a caller can't # silently pass encrypted=False alongside a target_key. copy_kwargs = { "SourceSnapshotId": source_id, "SourceRegion": region, "Encrypted": True if target_key else encrypted, } if target_key: copy_kwargs["KmsKeyId"] = target_key copy_id = ec2.copy_snapshot(**copy_kwargs)["SnapshotId"] mapping[source_id] = copy_id return mapping def wait_for_copies(ec2, copy_ids, poll_seconds=15, timeout_seconds=900): """Poll until every copy reaches a terminal state, or give up at the timeout.""" states, pending, deadline = {}, set(copy_ids), time.time() + timeout_seconds while pending and time.time() < deadline: resp = ec2.describe_snapshots(SnapshotIds=list(pending)) for snap in resp["Snapshots"]: if snap["State"] in ("completed", "error"): states[snap["SnapshotId"]] = snap["State"] pending.discard(snap["SnapshotId"]) if pending: time.sleep(poll_seconds) for copy_id in pending: states[copy_id] = "timeout" # never reached a terminal state in time return states def ready_for_delete(mapping, states): """Only originals whose copy is confirmed 'completed' are safe to delete.""" return [src for src, copy_id in mapping.items() if states.get(copy_id) == "completed"] Pitfall #2: a fixed-schedule cleanup system is only as good as its tag detection The mark stage typically excludes anything under active lifecycle management — snapshots tagged by AWS Backup, or by a snapshot lifecycle manager, since deleting those would fight a system that's already managing retention on its own schedule. The failure mode: the naive way to detect "is this managed by a lifecycle tool" is to check for specific custom tags that tool's policies are configured to set. The problem: lifecycle managers often also stamp a fixed, undocumented prefix on every resource they touch, regardless of what custom tags an individual policy configures — and if your detection logic only checks the policy-specific custom tags (built by enumerating currently active policies), you get a silent false negative the moment: a policy is deleted or rotated, but its old snapshots remain, or a policy's custom tag configuration doesn't match what your detection script expects. We found this the hard way: cross-referencing a "snapshots managed by X" report against a known-managed sample turned up a false negative on every snapshot we checked — each one came back "clear to delete" when it shouldn't have. The fix: stop trying to enumerate policies and instead just check for the lifecycle tool's own fixed, unconditional tag prefix directly — something like aws:dlm:* for AWS Data Lifecycle Manager — as a filter, independent of any specific policy's configuration. That prefix gets set on every snapshot the tool manages, full stop, with no way to configure it off. (The same idea applies if AWS Backup is in the mix too — check for its own unconditional tag prefix rather than its policy-specific tags.) The lesson generalizes: when you're trying to detect "is this resource managed by some other system," prefer checking for that system's own unconditional markers over reverse-engineering its current configuration. Configuration drifts and gets deleted; the tool's own fingerprint on the resource doesn't. Pitfall #3: the fan-out worker has a concurrency ceiling, and it's not documented up front The failure mode: even once the copy-before-delete ordering is fixed, there's a second limit that only shows up at scale — AWS caps the number of in-flight snapshot copies per destination region, per account, at 20. Fire off more than that concurrently and every excess request fails immediately with a ResourceLimitExceeded error — no queueing, no automatic backoff, just a hard rejection. Worth being precise about this one, since it's tempting to look for a quota-console fix: AWS's own Service Quotas listing marks "Concurrent snapshot copies per destination Region" as not adjustable. This isn't like a soft default you can raise with a self-service request — AWS raised the account-wide default itself from 5 to 20 in 2020, and some accounts have reportedly gotten a further increase through a support ticket, but neither path is something your pipeline can assume or trigger on its own. Batching and backoff aren't a workaround for a ceiling you just haven't asked AWS to raise yet — they're required regardless of what ceiling you end up with. This is easy to miss in testing because small batches never get close to the cap. It shows up the first time your fan-out worker reaches a large account: the first ~20 copies succeed, and everything after that starts failing in a burst — a reminder that "one independent unit of work per account" doesn't mean "no cross-account-shaped limits at all." The limit is per account/region, but it still governs how aggressively any single account's unit of work can run. The fix is external batching, since the copy engine itself typically exposes no concurrency knob: Split the eligible set into batches comfortably under the cap (e.g. 15, not 20 — leave headroom for anything else in the account that might also be copying snapshots concurrently, like an unrelated backup job). Sleep between batches long enough for in-flight copies to actually drain before the next batch fires — a copy that's still pending still counts against the limit. On ResourceLimitExceeded, back off and retry the same batch rather than aborting the whole run. Because the next attempt re-queries live eligible state instead of working off a stale list, this retry is safe: anything already copied-and-deleted in a partially-successful batch simply won't reappear as "still eligible." Expect large accounts to need much more conservative batch/sleep settings than small ones. A batch size and sleep interval that clears a 100-snapshot account instantly can still throttle hard against an 8,000-snapshot one — tune per account, not globally. Once you're safely under the concurrency cap, the sleep interval often stops being the bottleneck. Each batch's completion time is bounded by its slowest copy, and a same-region copy that also re-encrypts the data can take many minutes regardless of how short you make the sleep. At that point the only lever left for total wall-clock time is running independent scopes — separate regions, separate accounts — concurrently, since each has its own concurrency ceiling and doesn't compete with the others. This limit compounds with pitfall #1 above: a batch that partially fails with ResourceLimitExceeded mid-copy is exactly the kind of situation where "copy, then immediately delete" silently produces originals with no completed backup. Handling the concurrency ceiling and verifying copy completion are two separate fixes, and you need both. Putting pitfalls #1–#3 together, the shape of the fix looks roughly like this (pseudocode — the real thing is provider-SDK-specific, but the control flow is what matters): def process_account(account, batch_size=15, sleep_between=90): # Pitfall #2: filter out anything carrying another lifecycle tool's # own unconditional tag prefix (e.g. "aws:dlm:*") before it ever # reaches the eligible set below -- not just this policy's custom tags. eligible_snapshot_ids = get_eligible_snapshots(account, exclude_prefixes=["aws:dlm:"]) for batch in chunk(eligible_snapshot_ids, batch_size): copy_ids = {} for snapshot_id in batch: copy_ids[snapshot_id] = start_copy_with_retry( snapshot_id, max_retries=3, backoff_seconds=300, retry_on="ResourceLimitExceeded", ) states = wait_for_completion(copy_ids.values(), timeout_seconds=900) ready_to_delete = [ src for src, copy_id in copy_ids.items() if states[copy_id] == "completed" ] not_ready = [ (src, copy_id, states[copy_id]) for src, copy_id in copy_ids.items() if states[copy_id] != "completed" ] delete_originals(ready_to_delete) for src, copy_id, state in not_ready: log(f"{src}: copy {copy_id} ended in '{state}', leaving original in " f"place -- will retry next run") sleep(sleep_between) # let in-flight copies elsewhere drain before the next batch Three properties worth calling out: eligibility is filtered by fingerprint, not policy config (pitfall #2), retry targets a single batch, not the whole account (pitfall #3), and "not ready" never becomes "delete anyway" (pitfall #1) — it becomes a log line and a no-op, deferring to the next run's fresh eligibility check instead of forcing a decision on stale information. Pitfall #4: "skip AMI-linked snapshots" has a state your filter probably isn't checking The delete stage should obviously never touch a snapshot that's still backing an AMI — deleting it would silently break anyone's ability to launch from that image. Custodian ships exactly this as a built-in filter on aws.ebs-snapshot — type: unused excludes snapshots still referenced by an AMI — and it's tempting to treat that as a solved problem once it's in your policy. The failure mode: AMIs aren't just "exists" or "doesn't exist" — they also have a disabled state: still registered, still referencing its snapshots, but not launchable. This isn't a hypothetical gap either — it's a filed bug report against Custodian's own unused filter: it doesn't catch snapshots that are only referenced by a disabled AMI, as opposed to an active one. A filter that only checks for active/available AMIs will happily let a snapshot through if the only AMI referencing it has been disabled rather than fully deregistered. The snapshot matches every filter in your policy, gets copied, and then the actual DeleteSnapshot call fails at the AWS API level: An error occurred (InvalidSnapshot.InUse) when calling the DeleteSnapshot operation: The snapshot snap-xxxxxxxxxxxxxxxxx is currently in use by ami-xxxxxxxxxxxxxxxxx The unhelpful part: this exact message fires whether the blocking AMI is active or disabled. AWS doesn't distinguish the two in the error text, so there's no signal here that tells you "check disabled images too" — you have to already know that before you go looking, which is precisely the gap a filter that only checks active AMIs shares with a human reading this error for the first time. The compounding problem is what happens next if you don't catch this: the delete fails, but the copy that ran right before it already succeeded. On the next scheduled run, the probe step re-queries live eligibility, finds the same original still sitting there (never deleted), and processes it again — a fresh copy, another failed delete. Run that on a schedule for days without noticing, and you get a steadily growing pile of duplicate orphaned copies for the exact same handful of snapshots, none of which were ever going to delete successfully in the first place. It's a quiet, compounding waste: each occurrence looks like normal batch activity in the logs (a copy started, a delete attempted) unless you're specifically watching for the same source ID recurring across runs. The fix has two parts: Treat "in use by an AMI" as a state you verify independently of whatever built-in filter your policy engine provides, since a single boolean option may not cover every AMI state the underlying API cares about. Checking directly for any AMI (active or disabled) that references the snapshot as a block device mapping is the more defensive version. Add a check for recurring delete failures on the same source ID across runs, not just a check for failures in the current run. A snapshot that fails to delete once might be transient; a snapshot that fails to delete on every run for a week is a signal your filter has a gap, and it should stop being re-copied until that gap is understood. Concretely, that second point needs somewhere to keep count: log each failed delete as a structured line (source ID, reason, run ID) and put a metric filter on it — a CloudWatch Logs metric filter incrementing a per-source-ID counter works, or just a failure_count column in whatever state store your process_account loop already reads from. Either way, the check before copying is simple: if a source ID's count crosses a small threshold (three consecutive runs is enough to rule out a one-off throttle), skip it and alert instead of copying it again. The lesson generalizes from pitfall #2: when a cloud resource has a lifecycle with more than two states, don't assume a filter option that mentions the resource type has actually covered all of them. Read the error message the API gives you when something slips through — it's often more precise about the real constraint than the filter's documentation is. A few smaller gotchas Tag collisions between the three stages If your "copy" step preserves the source snapshot's tags on the copy (a reasonable default — you want the copy identifiable), watch out for reusing the same tag key across the mark → copy+delete → age-off stages. A copy of a snapshot that was already marked-for-deletion inherits that mark — including its already-past due date. If the age-off stage checks the same tag key, it can fire immediately instead of after the intended retention window, because the inherited mark is already expired. The fix is boring but effective: give each stage its own, never-reused tag key, even if the semantics feel similar. It costs nothing and eliminates an entire class of "why did this get deleted early" surprises. Marked-for-op state outlives the run that created it A fan-out worker scoped "by account" assumes each account's unit of work is self-contained per run. But if the delete stage's filter is simply "anything already tagged marked-for-op, past its date," it doesn't check which run wrote that tag — only that the tag exists. This showed up when an account got carved out of one rollout round and rescheduled into a later one, because its copy+delete step kept hitting the concurrency ceiling from pitfall #3 before the batching fix existed. By the time it came back around in the later round, that round's mark stage matched zero new snapshots there — everything eligible had already been marked earlier. Running the later round's delete stage against that account picked up the entire earlier backlog anyway, since the tag has no concept of "which round wrote me." Neither stage was wrong — mark and delete each did exactly what their filters said. The mismatch is between two assumptions that feel like they should line up but don't: "this account is in scope for this round" is an account-level idea, while "only this round's marks should act" is a tag-level idea the tag itself has no way to express. The fix here is a checklist item, not a code change: before pointing a delete stage at an account, ask what marked-for-op backlog could already be sitting there from an earlier, unrelated pass, and decide on purpose whether picking it up is correct — often it is, since it's still real work that needs finishing. If it isn't, isolate that account into its own run instead of folding it into a batch scoped for something else. Copies aren't the only place to put the safety net Not every retention window needs a live, same-region copy sitting around for the full period. If the safety window is long and access is expected to be rare, the EBS Snapshot Archive tier is worth considering as an alternative to Stage 3's copy — it's built for exactly this "keep it, but don't expect to touch it" case and can cut storage cost significantly versus a standard live copy. Takeaways Know which model you're actually running. A multi-stage tag-driven workflow (mark → act → age-off) and a fan-out worker across independent units of work (accounts, batches) are both easy to call "the pipeline" — but they fail differently, and conflating them hides where a given pitfall actually lives. "Sequential actions" in a policy engine is not the same as "this action waits for that one to finish." Verify intermediate state — like a copy actually reaching completed — before acting on it irreversibly. Detect by fingerprint, not by configuration. When checking whether a resource is managed by another system, prefer that system's fixed, always-on markers over trying to reconstruct its current policy config. Batch around concurrency limits, and make batches idempotent. Re-query the live eligible set before each batch rather than working off a stale precomputed list, so retries are safe and already-processed resources don't reappear. Tag-driven state doesn't know which run wrote it. Account-level scoping and tag-level filtering are two different notions of "in scope." Before pointing a later run at an account, check what backlog an earlier, unrelated run may have already left tagged there. A built-in filter option isn't proof a resource's whole lifecycle is covered. Custodian's unused filter sounds complete but only checks active AMIs; a disabled one still blocks the delete — this is a filed issue against the filter itself, not a hypothetical edge case. Watch for the same source ID failing repeatedly across runs — that's the filter telling you it has a gap, not just a transient error to retry. None of this required exotic tooling — just a healthy suspicion of "looks safe on paper" and a willingness to verify state with the cloud provider's own APIs before taking an irreversible action.