When Redis Goes Stale: Lessons on Cache Invalidation in an E-commerce System

On a Friday afternoon, the Sales team pinged the shared channel. A customer was complaining that a tour’s price on the website was almost 30% lower than the real one. The admin had changed the price that morning. Six hours later, the page still showed the old number.

I’m telling this story, and it’s not because Redis did anything strange. The fix turned out to live somewhere else entirely.

What happened

Our system uses cache-aside, the way everyone does it: a request asks Redis first, and on a miss it queries PostgreSQL and writes the result back to the cache with a one-hour TTL.

In theory a wrong price can’t live longer than an hour. Wherever the update happens, the cache expires on its own and gets refreshed.

In practice it lived six hours. Six times the TTL.

That number killed the “cache just hasn’t expired yet” theory. Something was writing the old price back into Redis after every expiry, and doing it consistently.

The culprit was a background job

I went through the logs and traced requests. It turned out to be a job that syncs content from Strapi, running every 15 minutes. It reads tours from the CMS and writes them to PostgreSQL. Then, as a side effect nobody paid attention to, it warms the cache with the exact data it just read.

Here’s the catch: the admin edits prices through our internal admin page, not through the CMS. Two different write sources, but only the CMS sync was allowed to write to the cache.

The sequence went exactly like this:

1. The old cache expires after an hour, as configured.
2. The next request misses. The service reads the new price from the DB and writes it to the cache. At this point the cache is correct.
3. Fifteen minutes later the CMS sync runs. It reads the tour from the CMS, where the price was never edited, and overwrites the cache with the old price.

sequenceDiagram
  participant U as Request
  participant S as Service
  participant R as Redis
  participant DB as PostgreSQL
  participant J as CMS sync (every 15 min)
  participant C as Strapi

  Note over R: tour cache expires after one hour
  U->>S: request tour price
  S->>R: read cache
  R-->>S: miss
  S->>DB: query new price
  DB-->>S: new price
  S->>R: write new price to cache
  Note over R: cache is correct now

  Note over J: 15 minutes later
  J->>C: read tour (price was never edited here)
  C-->>J: old data
  J->>DB: write to PostgreSQL
  J->>R: warm cache with the data it just read
  Note over R: old price is back in the cache

And it repeated. Every fifteen minutes, the cache was right for a few minutes and then wrong again.

This bug has a name: multiple writers writing to the same cache key, and no writer knows the others exist. Cache-aside assumes a single source of truth. Our system had two, out of phase with each other.

TTL couldn’t save this one

TTL is a safety net for the case where the cache was forgotten during invalidation. It doesn’t cover the case where the cache was invalidated wrongly, overwritten with stale data from another source.

TTL promises “the cache will eventually be correct”. It doesn’t promise “the cache is correct the moment the source data changes”. For a system that sells things, a wrong price means lost money or a dispute with a customer. “Eventually correct” isn’t enough.

The fix: only one channel may clear the cache

We dropped the “anyone can write the cache” model. In its place, a single invalidation channel, triggered by data-change events, no matter whether the change comes from the admin panel, the CMS sync, or a backfill script.

• Every service that updates price or inventory must publish a tour.price.updated event to Redis Pub/Sub instead of writing the cache itself.
• A dedicated subscriber listens, and it has exactly one job: delete the relevant cache key.
• The CMS sync job lost its permission to write the cache. It writes the DB and then publishes an event like every other source.

// After updating the price in the same DB transaction
async function updateTourPrice(tourId: string, newPrice: number) {
  await db.transaction(async (trx) => {
    await trx("tours").where({ id: tourId }).update({ price: newPrice });

    // Publish the invalidation event within the same transaction logic,
    // regardless of who calls this function (admin panel or CMS sync)
    await redisPubSub.publish(
      "tour.price.updated",
      JSON.stringify({ tourId, updatedAt: Date.now() }),
    );
  });
}

// Dedicated subscriber, the ONLY place allowed to delete the cache
redisPubSub.subscribe("tour.price.updated", async (message) => {
  const { tourId } = JSON.parse(message);
  await redis.del(`tour:${tourId}`);
});

Using Pub/Sub isn’t the interesting part. The interesting part is separating two questions cleanly: who may change the data, and who may delete the cache. Once there’s only one place that can delete the cache, the two write paths stop stepping on each other.

We kept the TTL, but it went back to its proper role: a second safety net, in case an event gets lost (Redis Pub/Sub doesn’t guarantee delivery) or the subscriber dies for a while. We cut the TTL from one hour to ten minutes. The cost is a few more DB queries, but the window of wrong data is much shorter.

The rules I kept

• Who is allowed to write the cache? If the answer includes more than one flow, that’s an early sign of exactly this kind of bug.
• Invalidate on events, not just on time. TTL treats “stale cache”. Event-driven invalidation treats “cache overwritten incorrectly”. Two different illnesses, two different doses.
• Background jobs are hidden writers. The CMS sync job was originally just for syncing content. At some point the cache-warming logic slipped in, nobody remembers when, and nobody ever reviewed its side effects. Any job that runs periodically and touches cached data should be re-examined periodically.
• Watch the drift between cache and DB. After the incident we added a job that compares a random sample of cached values against the DB and alerts on a mismatch. It’s cheap, and it’s the third layer of defense.

If I had to boil it down to one sentence: any system with more than one write path needs a clear invalidation strategy from day one. Shortening the TTL after every incident is not a strategy.