Day 25/60: Caching with Redis -- Cache-Aside, Stampede, and Stale Reads

60 Day Go Backend Engineering Challenge

Our GET /products/{id} endpoint read Postgres on every hit. During a promo the traffic concentrated on a few hundred hot SKUs, so the same SELECT ran thousands of times a second for identical rows. p99 was 180ms, the database sat near 80% CPU, and the shared connection pool was throttling everything else. A Redis cache-aside layer cut p99 to 9ms and dropped read QPS more than 95%. Then it shipped two classic bugs. This post is both.

Cache-aside in Go

The application owns the cache. Read Redis first; on a hit, return; on a miss, read the database, populate with a TTL, return. With go-redis, a missing key is the sentinel error redis.Nil -- a normal miss, not a failure. Any other error is Redis itself misbehaving, and you fall through to the database rather than failing the request.

type Cache struct { rdb *redis.Client db *pgxpool.Pool } func (c *Cache) Product(ctx context.Context, id string) (Product, error) { key := "product:" + id switch val, err := c.rdb.Get(ctx, key).Result(); { case err == nil: var p Product if json.Unmarshal([]byte(val), &p) == nil { return p, nil // hit } case err != redis.Nil: log.Printf("cache degraded, serving from db: %v", err) } p, err := c.fetch(ctx, id) // miss -> source of truth if err != nil { return Product{}, err } if b, e := json.Marshal(p); e == nil { c.rdb.Set(ctx, key, b, c.ttl()) } return p, nil } 

Choosing a TTL

The TTL is your tolerance for being wrong. It is the longest a value may be stale even if every other safety net fails, and it bounds memory by letting cold keys evict themselves. A product blurb tolerates minutes; a price or stock count wants seconds. Pick the longest staleness the feature survives, not a round number. And jitter it, so keys cached together do not all expire on the same tick:

func (c *Cache) ttl() time.Duration { // 10 min base + up to 60s jitter to de-synchronize expiries return 10*time.Minute + time.Duration(rand.Intn(60))*time.Second } 

The stampede

The first night, every hot-key expiry spiked p99 back to ~150ms. A hot key takes thousands of reads a second; the instant it expires, every concurrent request misses at once and runs the same query. The cache absorbed the load, then released it all in one synchronized burst -- a stampede, the thundering herd. Caching made it worse by aligning the misses perfectly.

Collapse concurrent misses into one DB call with singleflight. N callers for the same key share one execution; N-1 never touch Postgres.

import "golang.org/x/sync/singleflight" func (c *Cache) load(ctx context.Context, id string) (Product, error) { key := "product:" + id v, err, _ := c.sf.Do(key, func() (any, error) { p, err := c.fetch(ctx, id) if err != nil { return Product{}, err } if b, e := json.Marshal(p); e == nil { c.rdb.Set(ctx, key, b, c.ttl()) } return p, nil }) if err != nil { return Product{}, err } return v.(Product), nil } 

Jitter spreads expiries apart; singleflight handles whatever still lands together. Use both.

Invalidation on write

The quieter bug: a price update committed to Postgres but the page served the old value for ten minutes, because cache-aside never invalidates itself. Use delete-on-write -- change the row, then delete the key, in that order:

func (c *Cache) Update(ctx context.Context, p Product) error { if err := c.save(ctx, p); err != nil { return err // source of truth first } // Invalidate AFTER the commit. Delete, don't overwrite: // concurrent writers can race an overwrite into a wrong value. if err := c.rdb.Del(ctx, "product:"+p.ID).Err(); err != nil { log.Printf("invalidate product:%s failed: %v", p.ID, err) } return nil } 

Delete first and a failed write evicts a good value; worse, a concurrent read repopulates the stale row in the gap before your write lands. Commit, then invalidate.

Where it stops

Delete-on-write covers writes your service owns. It does not cover a manual SQL update or a second writer on the same table, and the delete is not transactional with the commit, so a crash in between leaves a stale key until the TTL saves you -- which is why the TTL stays mandatory. Stronger guarantees mean change-data-capture or write-through. For a read-heavy endpoint with one clear owner, cache-aside plus singleflight plus delete-on-write plus a sane TTL is the right amount of machinery.

A cache is a copy, and every copy can stampede and go stale. Plan for both on day one.