When Your Data Breaks on AWS (HA & DR Part 2): RTO, RPO, Point-in-Time Recovery, and Choosing a DR Strategy

In Part 1, we looked at what happens when a single bad write hits your database: how Aurora Backtrack can rewind it, how idempotency stops the same request from running twice, and why a restored database quietly leaves behind the data written during the restore.

Part 2 zooms out to the questions every team eventually has to answer:

• How much data can we afford to lose, and how long can we afford to be down?
• How does DynamoDB Point-in-Time Recovery actually work?
• Can we let automation detect a disaster and restore the database on its own?
• Which of the four AWS disaster recovery strategies should we pick?

I will be honest: some of my first answers to these questions were wrong. I will show you where I went wrong too, because those mistakes are exactly the ones that look smart on a whiteboard.

RTO and RPO: The Two Numbers That Decide Everything

Before you choose a single AWS service, you need two numbers. They are business metrics, not AWS features, and they shape your entire architecture.

RPO (Recovery Point Objective) is the maximum amount of data you can afford to lose, measured in time. If your database crashes at 12:00 PM and your RPO is 15 minutes, you must be able to restore the data as it was at 11:45 AM. Anything written between 11:45 and 12:00 is accepted as lost.

RTO (Recovery Time Objective) is the maximum amount of downtime you can afford. If the system crashes at 12:00 PM and your RTO is 4 hours, your team has until 4:00 PM to get everything back online.

A simple way to remember it:

• RPO looks backward: how far back in time is your last good copy?
• RTO looks forward: how long until users can use the system again?

Quick check: the business says "we can lose at most 5 minutes of data." Which metric is that? It is the RPO, because it is about data, not downtime.

Who decides RTO and RPO?

This surprised me. Engineers do not set these numbers. Business leaders, product managers and compliance officers do, based on how critical the system is and what the business can afford.

The engineer's job is to explain the cost of each choice. Take a healthcare system:

• Losing patient vitals is unacceptable, so the RPO needs to be close to zero.
• Downtime stops doctors from reading patient charts, so the RTO needs to be close to zero too.

Getting both near zero usually means running databases actively in more than one Region, which can cost thousands of dollars a month. If the business cannot pay for that, it has to accept a higher RTO and RPO. That is a business decision, made with the engineer's numbers in front of it.

What happens if you miss them?

Missing your RTO or RPO during a real outage is not just embarrassing:

• SLA breaches: if you promised customers a level of availability in a Service Level Agreement, you may owe them penalties or refunds.
• Regulatory fines: especially in healthcare and finance, where data loss and downtime are regulated.
• Reputation damage: customers remember the day your app lost their data.

DynamoDB Point-in-Time Recovery (PITR)

Now that we know what we are aiming for, let's look at one of the most useful recovery tools on AWS.

When you turn on Point-in-Time Recovery for a DynamoDB table, AWS continuously backs up every change your application makes to it. Every write, update and delete is captured as it happens, with no backup jobs for you to schedule.

Per-second granularity

Because every change is captured, you are not limited to a nightly backup from 2:00 AM. You can restore the table to its exact state at, for example, October 31, 2026 at 14:35:12. The most recent point you can restore to is usually about five minutes before the current time.

The rolling recovery window

PITR keeps a rolling window of up to 35 days (you can configure a shorter one). It works like a sliding window: as each new day is added, the oldest day falls off and is gone for good.

If you need to keep backups for months or years, for example for legal or compliance reasons, PITR is the wrong tool. Use AWS Backup or on-demand backups instead. Those are separate, full backups of the table that you can keep for as long as you choose.

PITR is turned on per table

DynamoDB is serverless, so there are no "instances" to configure. PITR is switched on table by table. You might enable it on your Orders table and leave it off for a temporary Sessions table to save money.

A restore never overwrites your table

This is the detail most people get wrong. When you restore with PITR, AWS never overwrites your original table. It creates a brand-new table (for example Orders-Restored) containing the historical data.

That is a safety feature. It stops you from accidentally destroying your current live data while trying to fix a mistake. To get the lost items back into the original table, you write a script that reads them from the restored table and copies them across.

One more thing to know: some table settings are not carried over to the restored table, such as auto scaling, CloudWatch alarms, tags and TTL settings. Plan to reapply them if you ever switch your app over to the restored table.

Can You Automate the Restore?

This was my favorite question, because I was convinced the answer was "yes, obviously."

Image suggestion: Upload icon-amazon-cloudwatch.png here (small, as a visual marker for the section).

Mechanically, you can. Here is how it would work:

1. A CloudWatch Alarm watches a metric, for example a sudden spike in write activity on the table, or a custom metric your application publishes every time it deletes something.
2. The alarm triggers an Amazon EventBridge rule.
3. EventBridge invokes a Lambda function.
4. The Lambda function calls the RestoreTableToPointInTime API.

In practice, you should almost never do this. There are two big reasons.

1. The function cannot know which second to restore to. The alarm fires at 10:05 AM. Should the function restore to 10:00? 10:04? If it guesses too late, the restored table still contains the corruption. If it guesses too early, it throws away valid user data.

2. False positives. Imagine a developer legitimately runs a cleanup script that deletes thousands of old test records. CloudWatch sees a massive spike, the alarm fires, and your automation kicks off a database restore nobody asked for.

"But what if we restore to the RPO time?"

This was my first attempt to fix problem number one. The idea: if our RPO is one hour, have the Lambda function restore to "now minus one hour". Lost data would always stay within the accepted limit.

It sounds clever, but it confuses a limit with a target. RPO is the maximum data loss you will tolerate, not the point you should restore to.

Here is what goes wrong. Your RPO is one hour. At 12:00:00 a bug starts deleting user profiles. At 12:05 the alarm fires and the function restores to "now minus one hour", which is 11:05.

You just threw away 55 minutes of perfectly valid data, everything from 11:05 to 12:00.

A human engineer would read the logs, see the bug started at exactly 12:00:00, and restore to 11:59:59. Valid data lost: zero.

"Then let the Lambda enforce the RTO"

You cannot enforce RTO with code. If you ask AWS to restore a 10 TB table, it might take hours simply because of how much data has to be copied. A Lambda function cannot make AWS finish faster. RTO is something you design for ahead of time, not something you can demand during an incident.

"We can mute the alarm during planned cleanups"

This part of my idea was actually right. Teams do this all the time. It is usually called a maintenance window or alarm suppression: before running a planned bulk cleanup, an engineer disables the alarm's actions, runs the script, then turns them back on.

"Can the alarm ignore deletes from the admin's IP address?"

My last idea was to make the alarm smarter: ignore mass deletes coming from a known administrator IP, and trigger recovery only when they come from anywhere else. It breaks down for three reasons.

CloudWatch metrics have no context. A DynamoDB metric is just a number, something like "500 writes in the last minute". It does not record who made those requests or from which IP address.

In a serverless app, DynamoDB never sees the user's IP anyway. The browser talks to API Gateway. API Gateway invokes Lambda. Lambda, running inside an AWS data center, talks to DynamoDB. From DynamoDB's point of view, every single request comes from the same Lambda function and its IAM role. A real customer and a rogue client look identical.

CloudTrail is too slow for this. AWS CloudTrail does record who called which API, but those events can take several minutes, sometimes up to 15, to reach EventBridge. By the time your automation reacts, a runaway script has been deleting data for a quarter of an hour.

The rule that actually works

Cloud engineers automate failover, but keep recovery manual.

• Failover is about availability (HA). If a whole Region becomes unhealthy, Route 53 can automatically send traffic to a healthy copy of your app in another Region. There is no judgment call: the healthy copy has the same data.
• Recovery from corrupted or deleted data is about accuracy (DR). A human needs to read the logs, find the exact second the bad change started, and trigger the restore to that point.

Prevent the Disaster Instead

Rather than building a robot that rewinds your database, the best teams make it very hard for the disaster to happen at all. Two habits do most of the work.

1. Least privilege IAM

Give each Lambda function only the permissions it actually needs. The function that handles customer orders needs to create and update items. It does not need to delete them, so it should not be able to.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowOrderWrites", "Effect": "Allow", "Action": [ "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem" ], "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/Orders" }, { "Sid": "BlockDeletes", "Effect": "Deny", "Action": "dynamodb:DeleteItem", "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/Orders" } ] } 

Even if this function's code goes completely wrong, AWS will refuse every delete it attempts. Only a separate admin process, protected with MFA, should be allowed to delete data.

2. Soft deletes

Instead of permanently deleting a record, add a flag like isDeleted. When a user deletes their account, the backend just flips the flag, and the frontend hides any record where it is true.

Before the user deletes their account:

{ "id": "user-4821", "name": "Amina", "email": "amina@example.com", "isDeleted": false } 

After:

{ "id": "user-4821", "name": "Amina", "email": "amina@example.com", "isDeleted": true, "deletedAt": "2026-11-06T12:00:00Z" } 

Now if a bug "deletes" 10,000 users, nothing is actually gone. An engineer runs a quick script to set isDeleted back to false for the affected records, and no database restore is needed at all.

Zooming Out: HA Inside One Region

Everything so far has been about data. Before we choose a full DR strategy, let's quickly cover the infrastructure side, because your first line of defense is High Availability inside a single Region.

An AWS Region is a geographic area with at least three Availability Zones (AZs). Each AZ is one or more separate data centers with their own power and networking, kilometers apart but within about 100 km of each other, linked by low-latency fiber.

The rule is simple: run production in at least two AZs. AZs protect you from a data center failure. Regions protect you from a regional one.

Three services do most of the work for HA inside one Region:

• Elastic Load Balancing runs health checks and only sends traffic to healthy targets.
• EC2 Auto Scaling replaces unhealthy instances and scales with demand.
• Amazon RDS Multi-AZ keeps a standby copy of your database in another AZ. If the primary fails, RDS fails over automatically, usually within one to two minutes, and the endpoint name stays the same, so your app does not need a new connection string.

The Four AWS Disaster Recovery Strategies

When the problem is bigger than one AZ, such as a Regional outage or a major data loss, you need a DR strategy that involves a second Region. AWS describes four, ranging from cheapest and slowest to most expensive and fastest.

Pilot Light vs Warm Standby

These two are the easiest to confuse.

• Pilot Light: your data is replicated and the core infrastructure is ready, but the application servers are off. It is cheaper, but those servers have to start and scale up before they can take traffic.
• Warm Standby: a smaller, fully working copy of the app is always running. It can take traffic immediately, then scale up to full size.

Tip: AWS Elastic Disaster Recovery gives you roughly pilot-light cost with recovery in minutes, because it keeps replicating your servers and can launch them in another Region quickly.

Replicate your data

A DR Region is useless if it has no data to recover with. These services copy your data across Regions:

• Aurora Global Database: copies data to other Regions at the storage layer, typically less than a second behind.
• DynamoDB Global Tables: multi-active, so you can write in every Region.
• S3 Replication: copies objects to a bucket in another Region.

But remember the lesson from Part 1: replication copies mistakes too. If someone corrupts data in your main Region, that corruption is replicated within seconds. That is why you need backups as well as replication, never just one of them.

The DR toolkit

Three services show up in almost every DR plan:

• AWS Backup: one backup policy for EC2, EBS, RDS, DynamoDB, EFS and more. It can copy backups to other Regions automatically, and Vault Lock stops anyone, even an admin, from deleting them early.
• AWS Elastic Disaster Recovery: continuously replicates your servers so you can launch them in another Region within minutes.
• Amazon Route 53: health checks plus failover routing, so users are sent to the healthy Region automatically.

How to Choose (and Why You Must Test)

The process is simpler than it looks:

1. Agree on RTO and RPO with the business for each system. Not every system needs the same targets.
2. Pick the cheapest strategy that meets those targets. If Backup and Restore meets your RTO and RPO, Active/Active is wasted money.
3. Automate the recovery steps with infrastructure as code (CloudFormation, CDK or Terraform), so rebuilding in another Region is a command, not a weekend of clicking.
4. Use both backups and replication.
5. Test it.

That last step is the one teams skip. My favorite line from preparing this session: "An untested DR plan is just a hope."

AWS gives you two tools for this:

• AWS Resilience Hub assesses your application against your RTO and RPO targets and tells you where you fall short.
• AWS Fault Injection Service breaks things on purpose, like stopping instances or adding latency, so you find out how your system really behaves before a real disaster does it for you.

Key Takeaways

1. RPO is how much data you can lose. RTO is how long you can be down. The business sets them, and engineers explain what they cost.
2. DynamoDB PITR lets you restore to any second in a window of up to 35 days, and it always restores into a new table, never over your original.
3. Automate failover, keep data recovery manual. A machine cannot reliably pick the right second to restore to, or tell a real incident from a planned cleanup.
4. RPO is a limit, not a target. Restoring to "now minus RPO" throws away valid data.
5. Prevention beats recovery. Least privilege IAM and soft deletes stop most data disasters before they start.
6. Pick the cheapest DR strategy that meets your RTO and RPO, keep both backups and replication, and test it regularly.

Wrapping Up the Series

Across these two articles we went from a single forgotten filter in a script to Region-wide disaster recovery. If there is one idea to take away, it is this: resilience is not one service you switch on. It is a set of decisions, made ahead of time, about how much you can lose, how long you can wait, and how you will prove your plan actually works.

If you missed it, Part 1 covers Aurora Backtrack, idempotency with Powertools for AWS Lambda, and the Delta Problem.

What does your team's DR plan look like, and when did you last test it? Tell me in the comments.