Transactional Outbox, Inbox & Consumer Idempotency
A dual-write commits the database and publishes as two steps, so one can succeed alone. The outbox makes the intent to publish part of the local commit. The inbox makes at-least-once delivery safe.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Question ladder
L1
Why is commit-then-produce unsafe?
Answer
A crash after the commit and before the produce leaves a row with no event.
L2
Why is produce-then-commit also unsafe?
Answer
A crash after the produce and before the commit leaves an event with no durable business row.
L3
What does the outbox put in the local transaction?
Answer
The business rows and an outbox row with a stable event id, type, and payload.
L4
What publishes the outbox row?
Answer
A relay. It can poll or it can tail the commit log. The implementation catalog lives on the transactional outbox lesson.
L5
Why is an inbox still required if the relay marks rows published?
Answer
The relay can observe success and crash before the mark, or the broker can redeliver. Side effects need their own dedupe.
L6
Which dedupe key fits a projection?
Answer
Apply the event only when its version is newer than the row. An inbox event id fits a charge or an email.
L7
How does this relate to a saga?
Answer
Saga steps that emit events should use the outbox so compensations are not chasing messages that never existed, or acting on ghosts.
Failure modes
Lost event
The business transaction committed and nothing was written that a relay can publish.
Ghost event
The broker has a fact and the business transaction rolled back.
Double side effect
The event was delivered twice and the consumer charged or emailed again.
Misconceptions
A database transaction and a Kafka transaction are one commit.
Ordinary application stacks do not share a coordinator across those systems. The outbox creates atomicity in the database, then a reliable relay.
Marking the outbox row published removes the need for an inbox.
Publish can still happen twice. Consumers dedupe before side effects.
Interviewer traps
Reciting SKIP LOCKED or the Debezium router from memory.
Name poll versus CDC at the architecture layer, then point at the transactional outbox lesson for the relay.
Saying Kafka exactly-once covers the charge.
Broker exactly-once narrows duplicate produces. Email, charges, and shipments still need an inbox.
The order and the fact share one commit
Prefer
Outbox row in the business transaction
Either the order and the intent to publish both persist, or neither does. A later relay hands the row to the broker. Consumers dedupe.
- Broker downtime is a backlog.
- A crash after commit still has a row to publish.
- A rollback emits nothing, so there is no ghost.
- The inbox absorbs a second publish.
Alternative
Commit, then produce
Two systems and no shared commit. One success and one failure is permanent drift until someone reconciles by hand.
- Crash after commit: the row exists, the event does not.
- Produce then roll back: consumers act on a fact with no row.
- Retries without a stable event id double-charge.
From local commit to a single side effect
The relay is a derived publisher. The consumer is where effects become safe.
- 1
One local transaction
Insert the business rows and an outbox row with a stable event id. - 2
Relay after commit
A poller or a change-data capture path publishes. Marking published happens after the broker accepts the record. - 3
At-least-once delivery
Assume duplicates. Broker transactions narrow producer duplicates. They do not finish the job. - 4
Inbox before the effect
The unique event id and the charge, email, or projection update commit together. - 5
Saga steps use the same glue
Compensations must not chase a message that was never durable. Undo policy stays on the sagas hub.
Overview
The classic event-driven footgun is the dual-write: update the database, then publish. Either step can succeed alone.
The transactional outbox makes "state change plus intent to publish" atomic inside one database. The inbox makes at-least-once delivery safe at the consumer.
This page is the architecture usage inside the cluster. Transactional Outbox & Inbox Patterns already teaches relay variants, including pollers and the Debezium outbox router. Do not rebuild that page. Change Data Capture owns log capture when the outbox table is streamed. Delivery semantics owns what the broker promises.
Both crashes
Sequence
- 1
Service → Database
1. Commit the order
- 2
Service → Broker
2. Produce OrderPlaced
- 3
Service
Failure A. Crash after the commit and before the produce. The event is lost.
- 4
Service → Broker
3. Produce first
- 5
Service → Database
4. Commit the order
- 6
Service
Failure B. Crash after the produce and before the commit. The event is a ghost.
Lesson map
One commit, or a ghost
The order and the outbox row are committed. The broker has not been published to yet.
Architecture. Service Committed. Database Committed. Broker Idle
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB service["Service Committed"] database["Database Committed"] broker["Broker Idle"] service -->|One commit| database database -->|Relay| broker broker -->|Inbox| service service -->|Lost event| broker service -->|Ghost| broker
Neither order is safe. There is no shared transaction between the database and the broker in a normal service. The fix is a single database commit that includes the event, and a publisher that only reads what committed.
Outbox, at the architecture layer
- In the same local transaction, write the business rows and an outbox row: event id, type, payload, headers.
- A relay publishes that row. It might poll, or a CDC connector might tail the table. Choose based on lag and who operates the connector.
- Mark the row published, or delete it, after the broker accepts it. A CDC relay can skip the mark and let the log be the cursor. That choice is operational, and it lives on the outbox and CDC lessons.
- Consumers process with an inbox or another dedupe rule.
| Relay style | What you are choosing | What you are not re-teaching here |
|---|---|---|
| Polling publisher | Simple and database-agnostic. Extra lag and a worker to run | Lock and batch SQL |
| CDC on the outbox table | Lower lag, less custom publisher code | Connector offsets, slots, and schema history |
| Database notify that wakes a publisher | Faster than a slow poll | The publisher still needs the outbox row for crash safety |
Cross-links: Transactional Outbox & Inbox Patterns for the implementation catalog. Change Data Capture when the relay is a change stream.
Inbox and other dedupe
At-least-once means duplicates. Pick a key that matches the effect.
| Strategy | Mechanism | Good for |
|---|---|---|
| Inbox table | Unique event id inserted with the side effect | Handlers that charge, email, or call a partner |
| Natural key | One row per order and action | Business-level dedupe when the event id is unstable |
| Upsert by version | Apply only when the event version is newer | Projections, as on the CQRS lesson |
| Broker exactly-once | Producer transactions inside a narrow scope | Fewer duplicate produces. Not a substitute for the rows above |
Delivery Semantics — At-Least-Once, At-Most-Once & Exactly-Once is the broker lesson. Your email, charge, and shipment still need the inbox.
Explicit outbox versus capturing the orders table
| Approach | Strength | Cost |
|---|---|---|
| Explicit outbox | Business-level events. You filter noise | The application must write the row |
| CDC on domain tables | No outbox write in the app | Consumers couple to table shape, deletes, and tombstones |
| CDC of the outbox table | Domain events plus a log-based relay | Outbox write plus connector operations |
Use the explicit event when the public fact is not one row image. Use table CDC when the downstream system should mirror the row. That fork is also on the CDC hub. This page only places it inside the event-driven map.
Outbox retry (run this)
The broker is down for the first relay attempt. The order and the outbox row are already stored. The second attempt publishes and clears the row. A dual-write would have lost the event on the first failure.
Press Run. Snippets must be self-contained — no network, files, or native modules.
The in-memory place_order stands in for one transaction. If you cannot name the single commit that holds both the order and the event, you still have a dual-write.
Inbox (run this)
The second delivery returns duplicate and does not append another charge. Production runs the inbox insert and the charge in one local transaction so a crash cannot record the effect without the key, or the key without the effect.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Sagas use the same boundary
A saga step that emits "stock reserved" or "payment captured" should write that event through the outbox. Otherwise a compensation runs against a message that never became durable, or a ghost message triggers an undo for a transaction that rolled back. The undo policy itself is Sagas & Distributed Transactions. This page only insists the message exists if and only if the local step committed.
Interview Q&A
Why not one transaction across the database and the broker?
Answer
They do not share a commit coordinator in ordinary application stacks. The outbox creates a practical atomic boundary in the database. The relay is asynchronous on purpose.
Is an inbox the same idea as an HTTP idempotency key?
Answer
Yes, at a different layer. Dedupe before side effects with a durable unique key. The HTTP lesson owns request keys. This page owns consumer keys.
What if the relay publishes twice?
Answer
Consumers must still be idempotent. The event id stays stable. The published mark should happen only after the broker accepts the record, and a crash between those two steps is a duplicate, not a new fact.
How does this relate to sagas?
Answer
Steps that emit events use the outbox so compensations are not chasing phantom messages. Sagas still define the undo.
Poll or CDC?
Answer
Poll when you want a simple worker and can tolerate its interval. CDC on the outbox when you want the commit log to be the publisher and you can operate the connector. Details stay on the outbox lesson and the CDC hub.
Does Kafka exactly-once remove the inbox?
Answer
No. It reduces duplicate produces inside a transaction scope. A charge, an email, or a shipment is your side effect. Delivery semantics draws that line.
What do you reconcile in production?
Answer
Outbox age and relay lag. A row that sits unpublished is the metric that replaces "we hope the produce happened."
When would you capture the orders table instead?
Answer
When consumers should mirror the row, including deletes. When they should see a domain fact with its own schema, write the outbox row.
Pitfalls
On a sketch of place-order, mark a crash between the database commit and the produce, then a crash between the produce and the commit. For each, say what the outbox version contains that the dual-write version does not.