With AWS FIS Lambda actions you can inject errors and delays into your Lambda functions without touching a single line of application code. The topic keeps me busy because almost every serverless application looks resilient on paper: retries here, a dead-letter queue there, alarms somewhere. Whether all of that actually works together in an emergency, you only find out once something has broken for real. Usually at night.
Chaos engineering turns this around. You break things on purpose, during the day, announced in advance and with an emergency stop. In this article you build a small, fully runnable example with the Serverless Framework: an API for a bike rental shop, a queue and a worker. After that you run two experiments and look at what happens.
In short
AWS Fault Injection Service (FIS) can inject errors, startup delays and fake HTTP responses into Lambda invocations through a Lambda extension that AWS maintains. The setup consists of a layer, two environment variables and an S3 bucket that FIS uses to distribute the configuration. The biggest pitfall is timing: while idle, the extension only checks once a minute, and the experiment duration, the queue’s visibility timeout and the retry count have to fit together, otherwise you are testing something other than you think.
What you will learn in this article
- How the FIS extension for Lambda works under the hood
- The three Lambda actions and what each one is good for
- A complete setup with Serverless Framework, alarms and experiment templates
- How to write a hypothesis and evaluate the result
- Typical mistakes, costs and safety rules for experiments
What is chaos engineering for Lambda?
Chaos engineering means: you state a testable assumption about your system, inject a controlled fault and compare the outcome with your assumption. It has little to do with breaking things at random. A good experiment is closer to a scientific trial, with a metric, a limited blast radius and a clear stop condition.
For Lambda this is particularly useful, because the interesting failures rarely sit in your own code. Things get exciting between the services: what does the SQS event source mapping do when the worker suddenly loses every third invocation? Does your client timeout hold when the API needs three extra seconds? Does the alarm fire at all? Those are exactly the questions an experiment answers.
A hypothesis instead of gut feeling
Every experiment starts with a sentence of the form “If X happens, Y stays within these limits”. For example: “If 30 percent of worker invocations fail for five minutes, no message ends up in the dead-letter queue, and all rental contracts exist no later than ten minutes after the experiment ends.” This sentence is measurable. That is why you can say afterwards, without any doubt, whether it was true.
How AWS FIS Lambda works under the hood
Since October 2024, AWS FIS has shipped its own actions for Lambda. Technically there is a Lambda extension behind them, which AWS provides as a public layer. An extension is an additional process inside your function’s execution environment. This one sits as a proxy in front of the Lambda Runtime API, the interface your runtime uses to fetch the next event and return the response. That lets it delay an invocation, mark it as failed or replace the response.
Communication runs through S3. When you start an experiment, FIS writes a small configuration file to a bucket you defined beforehand. The extension polls that location regularly. Without a running experiment this happens every 60 seconds by default, during an experiment every 20 seconds. That also explains why the fault does not start and stop to the second.
How AWS FIS injects faults through the Lambda extension: configuration via S3, polling every 60 or 20 seconds, alarm as emergency stop.
Per function, the setup needs four things: the layer, read access to the configuration folder, the variable AWS_FIS_CONFIGURATION_LOCATION with the S3 ARN of the folder, and AWS_LAMBDA_EXEC_WRAPPER set to /opt/aws-fis/bootstrap. The wrapper script starts the runtime in a way that makes it talk through the extension’s proxy.
Which Lambda actions does AWS FIS offer?
There are currently three actions. All of them take a duration in ISO 8601 format (for example PT5M for five minutes) and an optional share of affected invocations, invocationPercentage.
| Action | What it does | Typical question |
|---|---|---|
aws:lambda:invocation-add-delay | delays the start of the handler by startupDelayMilliseconds (default 1,000, maximum 900,000) | Do client timeouts and latency alarms hold? |
aws:lambda:invocation-error | marks the invocation as failed, with preventExecution even without running the code | Do retries, the DLQ and error alarms kick in? |
aws:lambda:invocation-http-integration-response | returns an HTTP status code and content type of your choice | How do callers behind API Gateway or an ALB react to a 500 or 429? |
The delay feels like a cold start, but it hits every execution environment and counts towards billed duration. If you set it higher than the function timeout, you also get a real timeout. That comes in handy when you want to check how your application behaves when a downstream service hangs.
The example: a bike rental shop with a queue
Our example is a small bike rental station. A client sends POST /rentals to an HTTP API. The createRental function validates the input, puts a message on an SQS queue and immediately answers with 202. The issueContract worker picks up the messages and writes the rental contract idempotently to DynamoDB. Whatever is left after five failed attempts ends up in a dead-letter queue (DLQ). You may already know the pattern from my article on the asynchronous API with Lambda.
🎬 Bike rental flow with injected faults – this animated diagram only runs on my site: watch it on sascha-ms.de
We want to run two experiments. The first makes 30 percent of worker invocations fail and checks whether the retries catch everything. The second delays half of the API invocations by three seconds and checks whether a client with a five-second timeout gets through without aborts.
Step by step: setting up AWS FIS Lambda with the Serverless Framework
Prerequisites
- an AWS account for testing or staging (not production) and credentials with permissions for CloudFormation, Lambda, IAM, S3, SQS, DynamoDB, CloudWatch and FIS
- Node.js 24 and npm
- Serverless Framework v4 (
npm i -g serverless, runserverless loginonce) - AWS CLI v2 to start the experiments
Step 1: create the project
Create an empty directory and install the dependencies. Serverless Framework v4 compiles TypeScript itself with esbuild during deployment, so you only need tsc for the tests and the load script.
mkdir radstation-chaos && cd radstation-chaos npm init -y npm install @aws-sdk/client-sqs @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb npm install -D typescript @types/node @types/aws-lambda { "compilerOptions": { "target": "ES2022", "module": "commonjs", "strict": true, "esModuleInterop": true, "skipLibCheck": true, "outDir": "dist", "rootDir": "." }, "include": ["src", "scripts", "test"] } Step 2: write the API function
The validation lives in its own file. That way you can test it without AWS.
// src/validate.ts export interface RentalRequest { bikeId: string; customerId: string; } export type ValidationResult = | { ok: true; value: RentalRequest } | { ok: false; error: string }; const ID_PATTERN = /^[A-Za-z0-9-]{3,40}$/; export function parseRentalRequest(body: string | undefined): ValidationResult { if (!body) return { ok: false, error: "Body is missing" }; let data: unknown; try { data = JSON.parse(body); } catch { return { ok: false, error: "Body is not valid JSON" }; } const { bikeId, customerId } = (data ?? {}) as Record<string, unknown>; if (typeof bikeId !== "string" || !ID_PATTERN.test(bikeId)) { return { ok: false, error: "bikeId is missing or invalid" }; } if (typeof customerId !== "string" || !ID_PATTERN.test(customerId)) { return { ok: false, error: "customerId is missing or invalid" }; } return { ok: true, value: { bikeId, customerId } }; } // src/create-rental.ts import { randomUUID } from "node:crypto"; import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs"; import type { APIGatewayProxyEventV2, APIGatewayProxyResultV2 } from "aws-lambda"; import { parseRentalRequest } from "./validate"; const sqs = new SQSClient({}); const queueUrl = process.env.RENTAL_QUEUE_URL!; export const handler = async ( event: APIGatewayProxyEventV2, ): Promise<APIGatewayProxyResultV2> => { const parsed = parseRentalRequest(event.body); if (!parsed.ok) { return { statusCode: 400, body: JSON.stringify({ message: parsed.error }) }; } const rentalId = randomUUID(); await sqs.send( new SendMessageCommand({ QueueUrl: queueUrl, MessageBody: JSON.stringify({ rentalId, ...parsed.value }), }), ); // 202: request accepted, the rental contract is created asynchronously return { statusCode: 202, body: JSON.stringify({ rentalId }) }; }; Step 3: write the worker
The worker reports failed messages individually (partial batch response) and writes with a condition, so a duplicate delivery does not create a second contract. I covered both in more detail in the articles on SQS partial batch response and idempotency in Lambda. For the experiment this matters, because repeated deliveries are exactly what we are about to provoke.
// src/issue-contract.ts import { DynamoDBClient, ConditionalCheckFailedException } from "@aws-sdk/client-dynamodb"; import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb"; import type { SQSBatchResponse, SQSEvent } from "aws-lambda"; const db = DynamoDBDocumentClient.from(new DynamoDBClient({})); const tableName = process.env.CONTRACT_TABLE!; export const handler = async (event: SQSEvent): Promise<SQSBatchResponse> => { const batchItemFailures: SQSBatchResponse["batchItemFailures"] = []; for (const record of event.Records) { try { const { rentalId, bikeId, customerId } = JSON.parse(record.body); await db.send( new PutCommand({ TableName: tableName, Item: { rentalId, bikeId, customerId, issuedAt: new Date().toISOString() }, // Idempotent: a second delivery of the same message creates no duplicate ConditionExpression: "attribute_not_exists(rentalId)", }), ); } catch (err) { if (err instanceof ConditionalCheckFailedException) continue; // already done console.error("Could not create contract", { messageId: record.messageId, err }); batchItemFailures.push({ itemIdentifier: record.messageId }); } } return { batchItemFailures }; }; Step 4: serverless.yml with layer and experiments
Now for the main part. You do not copy the extension’s layer ARN by hand; you take it from the public SSM parameter that AWS maintains per region. The Serverless Framework resolves ${ssm:…} during deployment. The environment variables apply to both functions here, because the whole stack is a staging stack. The file also contains the queues, the table, two alarms as stop conditions, the FIS role and both experiment templates.
# serverless.yml service: radstation-chaos frameworkVersion: "4" provider: name: aws runtime: nodejs24.x architecture: arm64 region: eu-central-1 stage: ${opt:stage, 'staging'} # FIS extension as a layer: the ARN comes from a public SSM parameter maintained by AWS layers: - ${ssm:/aws/service/fis/lambda-extension/AWS-FIS-extension-arm64/1.x.x} environment: AWS_FIS_CONFIGURATION_LOCATION: arn:aws:s3:::${self:custom.fisBucketName}/FisConfigs/ # Only set together with the layer – without it every invocation fails AWS_LAMBDA_EXEC_WRAPPER: /opt/aws-fis/bootstrap AWS_FIS_EXTENSION_METRICS: all # Recommended with preventExecution=true so the extension has a fresh configuration AWS_FIS_POLL_MAX_WAIT_MILLISECONDS: "2000" iam: role: statements: - Effect: Allow Action: s3:ListBucket Resource: arn:aws:s3:::${self:custom.fisBucketName} Condition: StringLike: s3:prefix: FisConfigs/* - Effect: Allow Action: s3:GetObject Resource: arn:aws:s3:::${self:custom.fisBucketName}/FisConfigs/* - Effect: Allow Action: sqs:SendMessage Resource: Fn::GetAtt: [RentalQueue, Arn] - Effect: Allow Action: dynamodb:PutItem Resource: Fn::GetAtt: [ContractTable, Arn] custom: fisBucketName: radstation-fis-${aws:accountId}-${sls:stage} functions: createRental: handler: src/create-rental.handler timeout: 10 environment: RENTAL_QUEUE_URL: Ref: RentalQueue events: - httpApi: method: POST path: /rentals issueContract: handler: src/issue-contract.handler timeout: 20 environment: CONTRACT_TABLE: Ref: ContractTable events: - sqs: arn: Fn::GetAtt: [RentalQueue, Arn] batchSize: 10 functionResponseType: ReportBatchItemFailures resources: Resources: FisConfigBucket: Type: AWS::S3::Bucket Properties: BucketName: ${self:custom.fisBucketName} PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true RentalDlq: Type: AWS::SQS::Queue Properties: MessageRetentionPeriod: 1209600 RentalQueue: Type: AWS::SQS::Queue Properties: VisibilityTimeout: 120 RedrivePolicy: deadLetterTargetArn: Fn::GetAtt: [RentalDlq, Arn] maxReceiveCount: 5 ContractTable: Type: AWS::DynamoDB::Table Properties: BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: rentalId AttributeType: S KeySchema: - AttributeName: rentalId KeyType: HASH # Stop condition 1: as soon as a message lands in the DLQ, the experiment stops DlqNotEmptyAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmDescription: Messages in the rental DLQ Namespace: AWS/SQS MetricName: ApproximateNumberOfMessagesVisible Dimensions: - Name: QueueName Value: Fn::GetAtt: [RentalDlq, QueueName] Statistic: Maximum Period: 60 EvaluationPeriods: 1 Threshold: 0 ComparisonOperator: GreaterThanThreshold TreatMissingData: notBreaching # Stop condition 2: more than 5 server errors per minute on the HTTP API Api5xxAlarm: Type: AWS::CloudWatch::Alarm Properties: AlarmDescription: 5xx errors on the bike rental API Namespace: AWS/ApiGateway MetricName: 5xx Dimensions: - Name: ApiId Value: Ref: HttpApi Statistic: Sum Period: 60 EvaluationPeriods: 1 Threshold: 5 ComparisonOperator: GreaterThanThreshold TreatMissingData: notBreaching FisLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: /aws/fis/radstation-${sls:stage} RetentionInDays: 14 FisExperimentRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: fis.amazonaws.com Action: sts:AssumeRole Policies: - PolicyName: fis-lambda-faults PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - s3:PutObject - s3:DeleteObject Resource: arn:aws:s3:::${self:custom.fisBucketName}/FisConfigs/* - Effect: Allow Action: - lambda:GetFunction - tag:GetResources Resource: "*" - Effect: Allow Action: - logs:CreateLogDelivery - logs:PutResourcePolicy - logs:DescribeResourcePolicies - logs:DescribeLogGroups Resource: "*" # Experiment 1: 30 % of worker invocations fail for five minutes WorkerErrorExperiment: Type: AWS::FIS::ExperimentTemplate Properties: Description: Bike rental – worker errors (30 %) RoleArn: Fn::GetAtt: [FisExperimentRole, Arn] Targets: WorkerFunction: ResourceType: aws:lambda:function ResourceArns: - Fn::GetAtt: [IssueContractLambdaFunction, Arn] SelectionMode: ALL Actions: workerErrors: ActionId: aws:lambda:invocation-error Parameters: duration: PT5M invocationPercentage: "30" preventExecution: "true" Targets: Functions: WorkerFunction StopConditions: - Source: aws:cloudwatch:alarm Value: Fn::GetAtt: [DlqNotEmptyAlarm, Arn] LogConfiguration: LogSchemaVersion: 2 CloudWatchLogsConfiguration: LogGroupArn: Fn::GetAtt: [FisLogGroup, Arn] ExperimentOptions: EmptyTargetResolutionMode: fail Tags: Name: radstation-worker-errors # Experiment 2: half of the API invocations start 3 seconds late ApiLatencyExperiment: Type: AWS::FIS::ExperimentTemplate Properties: Description: Bike rental – API latency (+3 s for 50 %) RoleArn: Fn::GetAtt: [FisExperimentRole, Arn] Targets: ApiFunction: ResourceType: aws:lambda:function ResourceArns: - Fn::GetAtt: [CreateRentalLambdaFunction, Arn] SelectionMode: ALL Actions: apiDelay: ActionId: aws:lambda:invocation-add-delay Parameters: duration: PT5M invocationPercentage: "50" startupDelayMilliseconds: "3000" Targets: Functions: ApiFunction StopConditions: - Source: aws:cloudwatch:alarm Value: Fn::GetAtt: [Api5xxAlarm, Arn] LogConfiguration: LogSchemaVersion: 2 CloudWatchLogsConfiguration: LogGroupArn: Fn::GetAtt: [FisLogGroup, Arn] ExperimentOptions: ...
