Email fixtures are easy to create and surprisingly hard to retire. A CI job allocates an inbox, sends a verification message, and exits. When the job is canceled, times out, or loses its final callback, the inbox can stay around forever. After a few weeks, the test system has a quiet collection of stale addresses, unread messages, and records nobody can confidently attribute to a pipeline run.

The usual response is to add a shorter TTL to every inbox. That helps, but it does not solve ownership. A TTL is a deadline; it is not evidence that the cleanup happened. In AWS CI/CD work, I prefer a small reaper process that treats test email data like any other temporary cloud resource: tag it, observe it, expire it, and report what could not be removed.

The hidden cost of test inboxes

An inbox used by an integration test has more than one resource behind it. There may be a provider-side mailbox, a row in the test database, an AWS secret reference, a Docker job, and an artifact containing the message ID. Deleting only one of those creates a split-brain cleanup state.

A fake emails generator is useful for isolated test data, but every generated address still needs an owner and expiry record.

I keep a simple inventory with an opaque inbox_id, run ID, commit SHA, environment, creation and expiry times, cleanup status, last error, and a secret_ref—never the secret value.

Define ownership before adding a TTL

The pipeline that creates the inbox should write the inventory record before it starts the application flow. If the test crashes between allocation and registration, the allocator should either support an idempotency key or return a provisional record that a later scan can discover.

I use the CI run ID as the idempotency key, but I do not use it as the only identity. Matrix jobs need a job or shard suffix too. A stable fixture strategy for matrix jobs is described in stable fixtures for matrix jobs; the same principle applies to email resources. Two jobs with the same commit should still have separate cleanup ownership.

The expiry policy should be explicit. A normal integration run may get 45 minutes after creation, while a failed run gets a 15-minute grace period. Do not guess from the latest message timestamp: a late message should not extend a failed run's lifetime.

A Docker cleanup worker for AWS CI

I package the reaper as a small Docker image and run it on a schedule, separate from application deployment. It reads only the inventory store and provider adapter; it does not need production database access or permission to send arbitrary mail.

The worker loop is intentionally boring:

set -euo pipefail docker run --rm \ -e INVENTORY_TABLE=ci-email-fixtures \ -e TARGET_ENV=staging \ -e DRY_RUN=false \ ci-email-reaper:latest reconcile --expired

In AWS, I run this from a scheduled task with a narrowly scoped role. It can read fixture inventory, fetch a short-lived adapter secret reference, delete test inboxes, and write cleanup status. It cannot read application secrets or alter deployment resources.

The adapter should make deletion idempotent. “Already gone” is a successful cleanup result, not an error that causes the worker to retry forever. Transient provider errors should be retried with a limit and recorded with a next-attempt time. Permanent errors, such as an invalid inbox ID, should move the record to quarantined so the same bad request does not fill the logs every minute.

Reconcile what the pipeline forgot

The reaper needs two scans. The first reads the inventory and handles records past expires_at. The second asks the provider for resources carrying the CI ownership tag and compares them with the inventory. That provider-side scan catches allocations that never made it into the database.

The comparison should be conservative. If a provider resource has an unknown owner, do not delete it immediately. Put it in an observation queue, attach a quarantine expiry, and require the environment policy to confirm that the resource belongs to CI. A missing database row is a reason to investigate, not a license for a broad delete.

The same boundary is useful in browser tests. A mailbox boundary in frontend tests prevents a test from reading another run's verification email; the inventory gives that boundary a durable owner.

I also keep a dry-run mode and print only opaque IDs. Never dump message bodies while diagnosing cleanup. Search notes sometimes contain phrases like tamp mail com or temp gamil com, but those labels are not an ownership system and should not be copied into logs or resource tags.

Metrics and failure handling

The useful metrics are operational:

  • expired fixtures found
  • deletion attempts and successes
  • resources discovered outside the inventory
  • age of the oldest non-quarantined fixture
  • cleanup failures by provider error class
  • records whose expiry was extended, with a reason

Alert on age and failure rate, not every retry. A provider outage can create many identical failures, while one orphaned inbox can sit unnoticed for days. Include the environment, opaque resource ID, and next action; never include a token or full email body.

If the reaper fails, the CI pipeline should not become green by accident. Keep cleanup status separate from the application test result, but make unresolved cleanup visible in the deployment report.

Operational checklist

  • Is every fixture tagged with environment, run, job, and expiry?
  • Is registration idempotent if the pipeline is interrupted?
  • Can the provider delete operation safely run twice?
  • Does a provider-side scan find resources missing from the inventory?
  • Are unknown resources quarantined instead of broadly deleted?
  • Are Docker credentials and AWS roles limited to the reaper's job?
  • Are message bodies, tokens, and secret values excluded from logs?
  • Is the oldest fixture visible on a dashboard or alert?

Q&A

Should cleanup happen inside every CI job?

The job should attempt cleanup in its final step, but a seperate scheduled reaper is still required. Cancellation and runner loss can skip final steps, so the pipeline cannot be the only owner of cleanup.

What should happen when deletion keeps failing?

Quarantine the record, preserve the provider error, and page only when the age or count crosses a policy threshold. A realible cleanup system makes failure visible without retrying a broken request forever.

Test email is temporary data, but temporary does not mean ownerless. With an inventory, a bounded Docker worker, and an AWS role designed for reconciliation, CI can clean up its own fixtures and leave a trustworthy trail when it cannot.