The idempotency check that has to run twice

Almost every guide to idempotent request handling says the same thing: give the operation a key, store the keys you have already processed, and check the store before doing the work. That advice is correct and it is also, on its own, not enough. The shelf-inventory movement code illustrates the gap and the second check used to address it.

The version that looks fine

The obvious implementation reads: look up the key, return early if it is present, otherwise take a lock on the row you are about to change, apply the change, record the key. Reviewers pass it. Tests pass, because a test sends the duplicate request after the first one has finished.

Now send the two requests at the same instant. Both look up the key and find nothing — neither has recorded it yet. Both proceed to the lock. One wins, does its work, records the key, commits. The second acquires the same lock a moment later and applies the identical movement again, because the only check it ever did happened before the lock, when the key genuinely was absent.

The check is not wrong. It is just answering a question about a moment that has passed by the time it matters.

The fix is one line and it is boring

Check again after acquiring the lock. With PostgreSQL Read Committed isolation, the next statement can see a preceding committed write: if another transaction recorded that key, you can now see it, and you return the earlier result instead of doing the work twice.

The first check is not redundant. It is the fast path, and it keeps the common case of an honest retry from queuing for a lock it does not need. The second check is the correct one. Keeping both is the point.

Why the shape of the operation matters more

The deeper lesson is that idempotency keys are a patch over operations that are not naturally repeatable. add 3 applied twice is wrong. set to 3 applied twice is fine, and needs no key at all.

So where I can, I write the operation as an assignment to an absolute value rather than a delta — reserve-to rather than reserve-more. Replays then converge instead of compounding, and the key becomes a defence in depth rather than the only thing standing between a retry and a wrong number.

Not every operation can be written that way. But it is worth asking before reaching for the key, because the reliable version of an unreliable network is mostly a matter of choosing operations that do not mind being repeated.

Filed under: concurrency, postgres