Distributed systems
Part 4 of 6 · Sagas & Distributed TransactionsCompensating Transactions — Idempotent Undo & Semantic Rollback
Idempotent semantic undo for sagas: reverse-order compensations, void versus refund, and reversing ledger entries.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
The charge already committed
Prefer
A new refund transaction
Payments have captured funds. Inventory has a hold. Compensation posts the inverse business effect and records that it did so, once.
- Other services may already have read the forward effect.
- A retry of the refund must not move money twice.
- The saga stays in compensating until every required undo finishes or is escalated.
Alternative
Roll back the distributed unit of work
There is no shared database undo log across inventory and the card network. The local transactions are already durable.
- ROLLBACK only exists inside a transaction that has not committed.
- Deleting the payment row hides the audit.
- A second refund attempt with a fresh random key double-pays the customer.
Rollback versus compensation
| Database rollback | Compensation | |
|---|---|---|
| When | Before the local commit | After the local commit |
| Who saw it | Other transactions did not keep the effects | Other services may already have reacted |
| Exactness | Bit-level undo inside that database | Business-level undo |
| If it fails | The transaction aborts | The compensation can fail and must escalate |
Garcia-Molina and Salem’s saga idea is this split: each local transaction commits, and each has a compensating transaction. The system is not isolated as one ACID blob while the saga runs. Design the business so those intermediate states are legal.
Design rules
- Name the inverse at design time. Every
dodeclaresundo, or you mark undo impossible and write the human or ledger path before you ship the step. - Idempotency key = saga id + step id + direction.
doandundodo not share a key. A retry of the refund is the same undo key. - Prefer cancel over refund when the payment step only authorized. Voiding an authorization is cheaper and less visible than capture plus refund.
- Order is part of correctness. Compensating out of order can corrupt state. Default to reverse completion order unless you have proved the operations commute.
- Audit matches the forward path. Refunds and cancellations are compliance records. An append-only ledger posts a reversing entry. It does not delete the capture.
Flow
- 1
1. Step N fails after local commits
- next2. Mark the saga COMPENSATING
- 2
2. Mark the saga COMPENSATING
- next3. Undo completed steps in reverse
- 3
3. Undo completed steps in reverse
- next4. Retry a transient undo with backoff
- 4
4. Retry a transient undo with backoff
- next5. Quarantine a poison undo
- 5
5. Quarantine a poison undo
- next6. Terminal FAILED or escalate
- 6
6. Terminal FAILED or escalate
Lesson map
Undo is a new transaction
The hold and the capture have both committed. The saga is COMPENSATING. Undo has not run yet.
Architecture. Saga log COMPENSATING. Inventory Committed. Payments Captured
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB saga["Saga log COMPENSATING"] inventory["Inventory Committed"] payments["Payments Captured"] saga -->|Reserve| inventory saga -->|Capture| payments saga -->|Refund| payments saga -->|Release| inventory payments -->|Poison undo| saga payments -->|Refunded| saga
Play walks the reverse undo. It is not a rule that every failure retries and then quarantines. A business decline skips hope-retries and goes straight to undo. A transient undo uses a bounded backoff inside the saga deadline. A poison undo stops the loop and pages. The state-machine lesson owns the timers. Retry Storms, Backoff & Jitter owns why the backoff is jittered. This page owns the inverse business action.
Strategies
| Strategy | Example | Naive risk |
|---|---|---|
| Release hold | Cancel an inventory reservation, or let a TTL expire | Double-release is safe only if release is idempotent |
| Void authorization | Cancel a card auth that was never captured | Void after capture is the wrong API call |
| Refund | Return captured funds | Partial amounts, currency, fees, and a second refund |
| Append-only ledger | Post a reversing accounting entry | Deleting the original row destroys the audit |
| Compensating message | “Ignore the earlier email” | Prefer delaying the send until the saga completes |
Irreversible side effects need a delay or a human path. You cannot unsend a shipment that left the building, and you cannot unsend an email the customer already read. If undo is impossible, the saga should not have performed that step until the steps that might fail are done, or it should enter a tainted state with a playbook. “Mark shipped” before “capture payment” is an invariant bug, not a compensation bug. The failure-modes lesson checks that invariant.
The undo event still needs a reliable publish
When you compensate, the local “refund row” and the fact PaymentRefunded commit together. That is the transactional outbox again. This page does not re-implement the relay or the inbox table. It requires them: a compensation that commits locally and then fails to publish leaves inventory and analytics believing the charge still stands.
Late success is the other race. A do that timed out may still capture. The undo and the late do must both check saga version or payment state before they apply. Otherwise you refund and then the delayed capture lands. The next lesson models that timeout. The guard itself is: do not apply a side effect if the saga is already in a state that forbids it.
Idempotent ledger (run this)
Capture and refund use different keys. A second capture does not add funds. A refund without a prior capture records the undo key and does not drive the balance negative. A second refund is a no-op.
Press Run. Snippets must be self-contained — no network, files, or native modules.
At a real payment provider the same rule is an idempotency key you store and send again. Generating a new random key on retry is how duplicate refunds happen. Stripe’s idempotent-request contract is the public version of this key.
Reverse-order undo (run this)
Completed steps come back in reverse. Shipping would undo before payment, and payment before inventory, when those steps had completed. This pair only completed reserve and pay, so the undo order is pay then reserve.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Interview Q&A
Why can’t we just roll back?
Answer
Other services already committed. There is no distributed undo log that spans them. ROLLBACK only works inside a transaction that has not committed. After commit, the inverse is a new transaction with its own failure mode.
What if compensation is impossible?
Answer
Say so in the design. Delay the irreversible step, keep a manual playbook, or mark the saga tainted until a person finishes it (a package that already left the dock). Do not pretend a compensating email deletes the first email.
How do you make refunds idempotent?
Answer
Use a stable refund key derived from the saga and the payment step. Store the provider outcome. Retries send that same key. Never mint a fresh key because the first call timed out.
Should compensation be synchronous?
Answer
Prefer a durable async undo with bounded retries so a process crash can resume. Synchronous undo is only for a short, reliable call. The user-facing copy should say cancelling until the saga reaches a terminal state.
What if a compensation fails partway?
Answer
Leave the saga in COMPENSATION_FAILED. Alert with the saga id and the undo log. Do not mark the business failed while money or stock is still stuck. Reconcilers in the failure-modes lesson finish the remaining undos.
How is this different from Try-Confirm-Cancel?
Answer
TCC cancel releases a hold placed by try, before the resource was finally taken. A classic compensation may undo a full commit, such as a refund after capture. Both are semantic inverses. The API you call depends on whether capture happened.
How do timeouts interact with undo?
Answer
A timed-out do can still succeed later. Undo and the late success must check state or a version so you do not refund and then accept a delayed capture. That race is why “timeout means it did not happen” is false.
How do you test compensations?
Answer
Fault-inject after each forward step. Assert the terminal saga state and the ledger invariant (balance, stock, no shipment without payment). Property-test retries of the same undo key. Include the “refund arrives before the forward capture is visible” case.
Pitfalls
For reserve, authorize, capture, and ship, write the undo call and the idempotency key. Mark any step whose undo is “human only.” Then crash the process after capture returns and before the saga row updates, and say which key the retry must send.