At a travel company, I worked on redesigning a monolithic email system into services. The APIs we built powered around 20 million personalised emails a day, both transactional and promotional, each one shaped by what the customer had searched for and booked.

This post is the design I'd draw today for that kind of system, worked through the way I'd present it in a design review. It isn't a description of that company's internals; it's the generalised version, with the lessons built in.

Requirements

Functional

  • Transactional email, triggered by events: booking confirmed, payment failed, trip tomorrow. Must arrive within seconds.
  • Promotional email, triggered by campaigns: a deal for an audience of millions. Must arrive within hours, and must respect each customer's preferences.
  • Personalisation: content based on search and booking history ("hotels in Goa, since you looked last week").
  • Compliance: unsubscribes are honoured immediately, and promotional email only goes to people who agreed to it.

Non-functional

  • 20M emails a day is about 230 a second on average. Averages are misleading here: campaigns go out in bursts. If 10M promotional emails go out in a two-hour window, that's roughly 1,400 a second for two hours, on top of transactional traffic. Design for the burst.
  • Transactional latency must not depend on what campaigns are running.
  • No email is sent twice, even when something retries.
  • The email provider has its own rate limits. Exceeding them, or sending to dead addresses, damages deliverability: the inbox providers start sending your mail to spam.

The design

Booking events and campaign runs both enter an eligibility step that checks consent, frequency caps and the suppression list. Eligible messages are rendered from a template plus a precomputed user profile, then split into a transactional lane that never waits and a rate-limited promotional lane, both sending through the email provider. Bounces and complaints come back through webhooks and update the suppression list.
Decide, personalise, send in two lanes, and close the loop. Explore the interactive diagram

1. Two kinds of trigger, one pipeline

Transactional emails start from events: the booking service publishes BookingConfirmed to Kafka, and the email platform consumes it. The booking service doesn't call the email service directly; if email is slow or down, bookings still work. (Publish those events with an outbox, or you'll occasionally lose one.)

Promotional emails start from a campaign run: a scheduled job resolves an audience (a query over customer data) into a stream of "send campaign X to user Y" requests.

Both become the same internal message, a send request: who, which template, why, and a priority. Everything downstream handles one shape.

2. Decide before you render

The cheapest email is the one you don't build. Before any personalisation, the eligibility step answers:

  • Consent: has this person agreed to this category of email? (Transactional mail about their own booking doesn't need marketing consent; promotions do.)
  • Suppression: have they unsubscribed, hard-bounced or complained?
  • Frequency caps: have we already sent them three promotions this week?
  • Quiet hours: is it 3am where they live? Then schedule it for later.

Doing this first matters for cost. At campaign scale, a significant share of an audience is often filtered out here. Every one of those is a render, a provider call and a quota slot you didn't spend.

3. Personalisation is a lookup, not a query

The naive version renders each email by querying the analytics store: "what did this user search for recently?" At 1,400 renders a second, against a warehouse built for large scans, that falls over.

Instead, precompute a profile per user: recent destinations, last search, loyalty tier, preferred language. A batch job writes it to a key-value store, and events keep the hottest fields fresh. Rendering then becomes template + one key lookup, which is fast and predictable.

The trade-off is freshness: a profile computed overnight doesn't know about a search from five minutes ago. For promotions that's fine. For anything triggered by a recent action, put the fresh data in the event itself (the booking event carries the booking) instead of looking it up.

4. Two send lanes, never one

This is the most important decision in the design. Transactional and promotional email get separate queues, separate workers and separate provider quotas.

With one shared queue, a 10-million-recipient campaign at 9am puts every booking confirmation behind it. The customer at the airport wondering whether their booking went through doesn't care that the queue is "working as designed".

With two lanes:

  • The transactional lane is small, always has spare capacity, and is scaled for latency.
  • The promotional lane is large, deliberately rate-limited to the provider's quota and a warm-up curve, and is scaled for throughput. When it's busy, it's just slower; nothing important waits.

Many email providers also let you send transactional and promotional mail from separate IP addresses or sub-accounts, so a campaign's spam complaints don't hurt the reputation of your password-reset emails. Use that.

5. Exactly one email, even with retries

Workers crash, provider calls time out, Kafka redelivers. Each of those can turn into a second copy of the same email, and customers notice duplicates.

Give every send request a deterministic message key, for example hash(userId, template, bookingId) for a booking confirmation or hash(userId, campaignId) for a promotion. Before sending, the worker records the key with a unique constraint; if the key already exists, it skips. Pass the same key to the provider as an idempotency key if it supports one. This is the same pattern as idempotency keys for payments, and it matters for the same reason: retrying must be safe.

6. Close the loop

The provider tells you what happened through webhooks: delivered, opened, bounced, marked as spam, unsubscribed. Two of these must change behaviour automatically:

  • A hard bounce (the address doesn't exist) goes on the suppression list. Repeatedly sending to dead addresses is one of the fastest ways to damage deliverability.
  • A spam complaint suppresses that person from promotional email immediately.

The rest (deliveries, opens, clicks) feed analytics, and back into the next profile computation.

Failure modes, and what happens

Failure Effect Design response
Provider is down Sends fail Requests stay in the queue; retry with backoff; transactional lane alerts within minutes
Provider rate-limits you 429 responses The promotional lane slows down; transactional quota is separate
Campaign audience is wrong Wrong people emailed Dry run first: count and sample the audience before the real run; kill switch per campaign
Worker crashes mid-batch Redelivery Message key prevents duplicates
Profile store is stale Less relevant content Acceptable for promotions; transactional data travels in the event
Template bug Broken emails at scale Render a sample to a seed list first; validate templates in CI

What I'd measure

A funnel per campaign and per template: requested → eligible → rendered → sent → delivered → opened, plus bounces and complaints. A sudden drop between two stages tells you where the problem is. For transactional email, also measure time from event to send, and alert on that rather than on queue length.

Takeaways

  • Separate transactional and promotional email completely: queues, workers and provider quotas.
  • Decide before you render. Consent, caps and suppression are cheap; rendering isn't.
  • Precompute personalisation so rendering is a lookup. Put fresh data in the triggering event.
  • Give every email a deterministic message key, so retries can't send twice.
  • Feed bounces and complaints back into suppression automatically. Deliverability is a feature.

Originally published at ayushwing.com.