Discussion on "Your Distributed Lock Expired. Your Code Didn't."

Patrick Kearns · 2026-10-13T07:44:23.559Z

A distributed lock can expire while the code that acquired it is still running. Another instance takes ownership, finishes its work and saves the result. Then the original instance resumes and overwri

The point that a longer expiry only delays the failure is the part people skip. The stale writer has no idea it lost the lease, so only the destination can refuse it.

The same thing shows up with agents that retry tool calls. A paused or timed-out run wakes up and writes after a newer run already did.

For side effects the database can't fence, like an email or a payment call, do you lean on idempotency keys with the provider, or an outbox so only the current owner ever sends?

Id prob combine an outbox with provider side idempotency. The fenced database transaction records the accepted result and its outbound intent together, so a superseded worker cant enqueue new work. The dispatcher then retries that intent using a stable idempotency key for the logical operation. An outbox doesnt guarantee "only the current owner ever sends". An intent accepted before ownership changed can still be delivered afterwards, and a dispatcher can itself pause with a request in flight. If that previously accepted action must now be rejected as obsolete, the destination needs a version or conditional operation check too. Idempotency alone wont establish that. Where the provider offers no check, you have to decide whether accepted intents remain valid or whether the workflow needs cancellation, reconciliation or compensation.

Good distinction between fencing for safety and lease expiry for liveness. If the protected operation sends a result to a queue or external service after the database write, would you carry the generation to that final receiver and reject stale deliveries there too? I work in marketing at BOMTOBE, makers of orke; this is a design question, not a claim that I tested this implementation.

Yeah, f the downstream operation needs protection against stale owners, Id carry the report ID and generation through and have the receiver enforce the check atomically with its own state change. The database fence only protects the database write, it wont automatically protect a later queue message or API call.

One thing worth adding for anyone adopting this: count rejected publications by cause. Zero rows because Generation moved means another worker took over, which is the design working. Zero rows with Generation unchanged means only the lease time ran out, so builds are outliving the lease and its length or renewal needs attention. The UPDATE alone cannot tell the two apart, so a follow-up read of the row is needed to log which one happened. Did you consider returning the reason from the adapter instead of a bool?

Agreed—the reason should be visible, not collapsed into one ‘rejected’ count. For a downstream queue, the generation token only prevents a stale publish if the receiver checks it; otherwise an idempotency key or reconciliation step is still needed. Does the sample report track these rejection causes separately?

No, the sample currently logs a generic publication refusal with the report ID and generation, it doesnt track causes separately. A structured result would be a useful extension, provided it preserves the distinction between a reason and a later diagnostic observation. The thing to remember with a downstream receiver is an idempotency key prevents duplicate execution of the same operation, but doesnt necessarily reject a distinct operation from a stale owner. The receiver needs to enforce the generation check atomically with the protected change. Otherwise it wont provide that same prevention guarantee.

Fair point about another acquisition landing between the UPDATE and the read. Logging it as diagnostic evidence with an unknown outcome allowed seems like the honest version, and it keeps the adapter contract to what it can actually guarantee. Thanks for the detailed answer.

I think the publish statement can drop AND LeaseUntilUtc > SYSUTCDATETIME() without losing safety, because the generation already fences the write. If nobody acquired after the lease ran out, Generation still matches and LeaseOwner is still the same worker, so the late write can't overwrite anyone. If someone did acquire, Generation has moved and the update matches zero rows either way. As written, a build that takes 31 seconds with no competing worker throws its result away, which is exactly the case the missing renewal loop would have to cover.

Kleppmann's 2016 locking post draws the same line: safety shouldn't rest on timing, and timeouts belong to liveness. Here the lease decides when another worker may take over, and the generation decides whose write counts. If you want a cap on how old a published report can be, I'd keep that as a separate freshness rule rather than part of the fence.

The post deliberately applies the stricter rule that publication requires an unexpired lease, but your 31-second example highlights the cost of that choice, especially without renewal. For this report example, allowing publication until the acquisition is superseded is a better fit. Id remove that predicate and update the explanation and expiry test accordingly, keeping any source freshness requirement separate.

Search Hashnode

Your Distributed Lock Expired. Your Code Didn't.

Responses11