Debugging a Hidden Race Condition in a Real-Time Payment System

There’s one kind of bug I hate more than any other. The business logic isn’t wrong. What’s wrong is the tiny window between two operations that look sequential.

This case was a race condition. A few bookings were recorded as paid twice.

What we saw: one order, two charges

Operations reported that some tour bookings had status paid, but the amount in the transactions table was double the order value. Not every booking, about 0.3% of them.

Small. But enough to cost customers’ trust and make accounting reconcile by hand.

The payment handling logic at the time looked airtight:

async function handlePaymentWebhook(payload: PaymentWebhookPayload) {
  const order = await orderRepo.findById(payload.orderId);

  if (order.status === "paid") {
    // Already processed, skip
    return;
  }

  await orderRepo.markAsPaid(order.id, payload.amount);
  await ledgerRepo.recordTransaction(order.id, payload.amount);
}

At a glance it’s safe: check the status first, only process if it isn’t paid yet. So where’s the bug?

Exposing the race condition

The payment provider we integrate with (one of the Adyen/Stripe gateways) retries a webhook if it doesn’t get a 200 response within a short window. A slow network, or the service being busy with other requests, is enough. The provider fired two webhooks almost simultaneously for the same payment event.

Two handlePaymentWebhook calls then ran in parallel. What actually happened:

1. Request A reads order.status and sees "pending".
2. Request B reads order.status and also sees "pending", because A hasn’t written the new status yet.
3. Request A writes status = "paid" and writes a transaction.
4. Request B still thinks the order is unpaid, writes status = "paid" again (harmless), and writes a second transaction. The ledger now has two money rows for one order.

On a timeline it’s compact:

sequenceDiagram
  participant P as Payment provider
  participant A as Request A
  participant B as Request B
  participant D as Database

  P->>A: webhook 1
  Note over P: no 200 response in time, retries
  P->>B: webhook 2, sent almost simultaneously
  A->>D: read order.status
  D-->>A: "pending"
  B->>D: read order.status
  D-->>B: "pending"
  A->>D: write status = paid, write transaction
  B->>D: write status = paid, write second transaction
  Note over D: one order, two money rows

This pattern has a name: check-then-act. You check, then you act, and nothing guarantees nobody sneaks in between the two steps. In a single-threaded app the code looks perfectly reasonable. But with multiple service instances running in parallel, or even just Node.js interleaving async requests, the gap between read and write is enough for two flows to both pass the check.

Debugging a race condition: rebuilding the timeline from logs

A race condition doesn’t reproduce reliably, so it’s hard to catch. Here’s how I went about it:

1. Log correlation by orderId and requestId: log when the webhook arrived, when the status was read, when it was written, all with millisecond timestamps. With those logs I could compare timelines across the broken bookings, and the gap between the two webhooks turned out to be only about 40-120ms.
2. Check the payment provider’s logs: confirm that two webhooks really were sent for the same payment_intent_id, and that we weren’t accidentally calling ourselves twice.
3. Reproduce locally by firing 2 requests at once: a small script calling Promise.all with two identical fake webhook requests against the same order in pending state. A few runs were enough to reproduce the bug exactly, which confirmed the race condition hypothesis.

The fix: push the constraint down into the database

if (order.status === "paid") return; in the application layer isn’t enough, because it isn’t atomic. Between the read and the write, another flow has already changed the status. The only place that can truly guarantee uniqueness is the database.

We added two layers.

Layer 1: a unique constraint on the ledger table

The constraint is on orderId + providerEventId, the unique event ID the payment provider attaches to every webhook:

ALTER TABLE transactions
ADD CONSTRAINT uniq_order_provider_event
UNIQUE (order_id, provider_event_id);

With it, even if two webhooks both get past the check in the application layer, the second write into transactions is still rejected by the database with a unique violation. The code just has to catch that error and treat it as already handled:

async function handlePaymentWebhook(payload: PaymentWebhookPayload) {
  try {
    await db.transaction(async (trx) => {
      await trx("transactions").insert({
        orderId: payload.orderId,
        providerEventId: payload.eventId,
        amount: payload.amount,
      });
      await trx("orders").where({ id: payload.orderId }).update({ status: "paid" });
    });
  } catch (err) {
    if (isUniqueViolation(err)) {
      // Duplicate webhook, already handled by an earlier call — safe to skip
      return;
    }
    throw err;
  }
}

Layer 2: an advisory lock per orderId

For more complex read-modify-write operations, such as accumulating the remaining seats of a tour, a simple unique constraint can’t express the whole business logic. In those cases I use an advisory lock:

await db.transaction(async (trx) => {
  // Take an advisory lock on orderId, released automatically when the transaction ends
  await trx.raw("SELECT pg_advisory_xact_lock(hashtext(?))", [payload.orderId]);

  const order = await trx("orders").where({ id: payload.orderId }).first();
  if (order.status === "paid") return;

  await trx("orders").where({ id: payload.orderId }).update({ status: "paid" });
});

pg_advisory_xact_lock lets only one transaction work on a given orderId at a time. Any other transaction waits until the lock holder commits or rolls back. An in-memory variable or a Node.js mutex can’t do this when several service instances run in parallel. An advisory lock can, because PostgreSQL manages it centrally.

What I kept from the race condition case

• An idempotency key isn’t always generated by the client. For webhooks, providerEventId plays that role. Leaning on it at the unique-constraint level is much cheaper than building a separate idempotency mechanism.
• Check-then-act in the application layer is never atomic if it can run in parallel. The constraint has to live at a layer that can guarantee atomicity, usually the database.
• Assume webhooks arrive duplicated and out of order. That’s the default assumption when integrating with any payment provider, not an edge case.
• Running things sequentially won’t reproduce a race condition. To see it, you have to deliberately create production-like parallelism: Promise.all, or a load-test tool firing concurrent requests.

The 3 AM deadlock incident from before also started with a request being sent again. Different road, same dead end.

One question I still can’t answer: how many times a day does the provider retry? The current code catches the unique violation and moves on, so that number isn’t recorded anywhere.