Transactional Outbox & Inbox Patterns
Dual-write is the bug. Write domain row + outbox row in one DB transaction; poller/CDC publishes; inbox dedupes at-least-once deliveries into effectively-once processing.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Why the outbox beats dual-write
Prefer
Domain row + outbox row in one ACID transaction
Either both persist or neither. A poller or CDC publishes PENDING rows. Crash after commit cannot lose the event.
- No distributed transaction between Postgres and Kafka.
- Broker downtime is a backlog, not a lost order.
- Outbox rows are an audit log you can replay.
- Inbox ON CONFLICT on the far side eats publish duplicates.
Alternative
Write the DB, then produce to the broker (dual-write)
Two commits, two failure domains. One succeeds; the other does not. You will not notice until reconciliation.
- Crash after INSERT order, before produce: order exists, no event, downstream never bills.
- Crash after produce, before COMMIT: event exists, order rolls back — phantom work.
- Retrying the whole handler without a key double-inserts.
- Kafka EOS does not include your earlier SQL session.
Happy path — outbox then inbox
Vertical cards for the request, the relay, and the inbox skip.
- 1
BEGIN
One connection, one transaction. No produce inside this tx. - 2
INSERT order + INSERT outbox PENDING
Same COMMIT. Payload, topic, partition key, sequence_id live on the outbox row. - 3
Return 202 to the client
The event is durable even if Kafka is down. Do not block the request on produce. - 4
Poller or CDC publishes
SKIP LOCKED so many workers can drain. On ack, mark SENT. On produce fail, leave PENDING. - 5
Consumer inbox insert
INSERT inbox(msg_id) ON CONFLICT DO NOTHING. Only the winner runs the handler. - 6
Duplicate delivery
Second insert affects 0 rows. Skip. At-least-once on the wire, effectively-once in the service.
Overview
Microservices want a state change and a message for the rest of the world. Those live in two systems. Two COMMITs is the dual-write problem:
- DB succeeds, broker fails → silent skip. The order is paid; fulfillment never hears.
- Broker succeeds, DB rolls back → ghost event. Fulfillment ships a row that does not exist.
- Both retry independently → duplicates on one side, gaps on the other.
The transactional outbox puts the message in the database that already owns the truth. One ACID transaction writes:
- The domain row (
orders). - An outbox row (
event payload,topic,key,status = PENDING).
A poller or CDC process reads committed outbox rows and publishes to the broker. The publisher is at-least-once (it may send twice if it crashes after send, before SENT). Downstream uses an inbox to ignore the second copy.
Write side: exactly-once relative to business state (atomic with the row).
Read side: at-least-once from the broker → effectively-once via inbox.
That is the whole pattern. Kafka EOS is complementary, not a replacement: EOS coordinates broker produce + offset, not your INSERT INTO orders.
Why it matters
- Integrity. Order and
order_createdcannot diverge afterCOMMIT. - Scale. The request path does not wait on Kafka. Pollers scale with
SKIP LOCKED. - Reliability. Broker outage = outbox lag, not lost writes.
- Observability. Outbox rows are the audit trail. Replay is
WHERE status = SENT AND created_at ….
Dual-write vs outbox
Flow
- 1
1. Dual-write commits the order
- next2. Produce is a second step
- 2
2. Produce is a second step
- next3. Crash before send loses it
- next4. Send then rollback is a ghost
- 3
3. Crash before send loses it
- 4
4. Send then rollback is a ghost
- 5
5. BEGIN one database transaction
- next6. INSERT order and outbox
- 6
6. INSERT order and outbox
- next7. COMMIT both or neither
- 7
7. COMMIT both or neither
- next8. Poller or Debezium relay
- 8
8. Poller or Debezium relay
- next9. Broker delivers at least once
- 9
9. Broker delivers at least once
- next10. Inbox skips the duplicate
- 10
10. Inbox skips the duplicate
Lesson map
Transactional Outbox & Inbox Patterns
Dual-write is the bug. Write domain row + outbox row in one DB transaction; poller/CDC publishes; inbox dedupes at-least-once deliveries into effectively-once processing.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB a1["1. Dual-write commits the order"] a2["2. Produce is a second step"] a3["3. Crash before send loses it"] a4["4. Send then rollback is a ghost"] a1 -->|1. Dual-write commits the order| a2 a2 -->|2. Produce is a second step| a3 a2 -->|2. Produce is a second step| a4
Outbox table (logical)
outbox
id BIGSERIAL PRIMARY KEY
event_id UUID UNIQUE -- idempotent publish key
aggregate_id TEXT NOT NULL -- order_id; partition key
sequence_id BIGINT NOT NULL -- per-aggregate order
topic TEXT NOT NULL
payload JSONB NOT NULL -- versioned envelope
status ENUM('PENDING','SENT','FAILED')
created_at TIMESTAMPTZ NOT NULL
sent_at TIMESTAMPTZ
attempts INT NOT NULL DEFAULT 0
INDEX (status, created_at) -- poller
UNIQUE (aggregate_id, sequence_id) -- orderingPoller vs CDC
| Publisher | How | Pick when |
|---|---|---|
| Poller | SELECT … FROM outbox WHERE status = 'PENDING' ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT n then produce, then status = SENT | Simple; OK for seconds of lag; works when you cannot run Debezium |
| CDC (pgoutput, Debezium outbox event router) | Streams the insert from the WAL. No polling SQL on the primary (or much less). | Low latency; already have Kafka Connect |
SKIP LOCKED is the multi-worker trick: a row locked by poller A is invisible to poller B, so you scale drainers without double-publish from two live workers. You still double-publish if A sends, dies before SENT, and B picks the row up — that is why the consumer has an inbox.
CDC does not magically exactly-once the broker either. Debezium can emit the same WAL record twice across connector restarts. Inbox still required.
Deep dive · Do not produce inside the web request
Blocking POST /orders on kafka.produce().get() re-introduces dual-write (or at least dual-latency). Commit the outbox, return 202 Accepted, let the worker publish. The client already has idempotency keys for the HTTP call; the outbox is the async counterpart.
Table CDC versus a domain outbox
Log-based CDC on orders mirrors storage. Every column change becomes an event. That is the right contract for a search index, a cache invalidation, or a warehouse table that should look like orders. It is the wrong contract for OrderPaid.
A payment can update orders, insert payments, and reserve inventory in one database transaction. Table capture emits three noisy row events, in commit order, on three keys. None of them is the public fact. The public schema also tracks every OLTP rename and every column you did not mean to publish, including PII sitting in before / after images.
| Table CDC | Outbox row | |
|---|---|---|
| Event meaning | A row mutation | An explicit domain intent |
| Schema | Coupled to OLTP columns | Payload versioned on its own |
| Multi-table business transaction | Many events | One outbox row, or a few you chose |
| PII | Hard to keep out of the envelope | Emit only what consumers need |
| Projection rebuild | Natural from row history | Needs event versions or snapshots |
Use table CDC when the downstream copy should follow the table. Use this outbox when the event is the workflow. Capture mechanics (slots, snapshots, heartbeats) are Debezium & Kafka Connect. The decision to capture at all is Change Data Capture. This page stays the idempotency lesson: one commit, then a relay, then an inbox.
Debezium outbox router versus the poller
The relay is not a second transaction with the business write. It reads a commit that already happened.
Debezium outbox event router (preferred when Connect is already there).
- The request
BEGINs, writes the business rows,INSERTs the outbox, andCOMMITs. Still no produce in the request. - Debezium reads the outbox insert from the WAL. That is log-based capture of one table, not a poll of
PENDING. Slot, snapshot, and offset flush behavior is the Debezium lesson. A connector restart can emit the same insert twice. - The Outbox Event Router SMT takes
aggregate_typeandevent_typeonto the topic you configured, and usesaggregate_idas the message key. The SMT does not create atomicity. The database commit already did. - The consumer opens its own transaction, inserts the inbox, and applies. Delivery is at-least-once. Effects are effectively-once only with that inbox, or with an LSN guard when the sink is a projection. Sink patterns are Exactly-Once CDC Pipelines.
Flow
- 1
1. BEGIN business writes plus outbox
- next2. COMMIT one transaction
- 2
2. COMMIT one transaction
- next3. WAL holds the outbox insert
- 3
3. WAL holds the outbox insert
- next4. Debezium reads the slot
- 4
4. Debezium reads the slot
- next5. Outbox SMT routes by type
- 5
5. Outbox SMT routes by type
- next6. Topic keyed by aggregate id
- 6
6. Topic keyed by aggregate id
- next7. Inbox dedupes then applies
- 7
7. Inbox dedupes then applies
Polling relay. SELECT … FOR UPDATE SKIP LOCKED still works, and it is the right tool when you cannot run Connect. It has query-CDC costs: poll interval, primary reads, and a mark-published race. Those tradeoffs are WAL Tailing vs Query-Based CDC. SKIP LOCKED stops two live workers from claiming the same row. It does not stop a crash-after-send from publishing twice. The inbox still required.
LISTEN/NOTIFY plus a read. The notify wakes a worker quickly. Notifications are not durable. A missed notify must fall back to a poll or to CDC. Do not treat the notify as the source of truth.
| Relay | What you gain | What you still owe |
|---|---|---|
| Debezium outbox SMT | Low lag, no poll load, capture is the commit | Connect ops, SMT config, at-least-once offset flush |
| App poller | Simple deploy, no log privileges | Poll lag, delete-or-mark strategy, primary load |
| LISTEN/NOTIFY | Fast wake-up | Loss of the notify; keep a poll or CDC backup |
If the relay is down, the outbox table grows and business commits still succeed. Backpressure is lag, not a lost order. If the relay is Debezium and the connector stops, the replication slot also pins WAL. That disk risk is CDC Failure Modes. It is not a reason to go back to dual-write.
2PC / XA across Postgres and Kafka is the theoretical shared commit and the operational one to refuse: coordinator failure, lock duration, heuristic outcomes. “Publish after the orchestrator writes” is still dual-write unless the write includes this outbox. “Listen to the table change feed only” is fine for projections and weak for a stable public event.
Routing key and payload size
Per-entity order uses aggregate_id as the bus key. Global order is not free. How a key selects a partition stays in Partition Keys — Ordering Guarantees vs Parallel Throughput. Do not re-teach it here. Say the key, then stop.
Huge JSON in the outbox is huge WAL once CDC is tailing that table. Prefer a skinny event or a URI to the body. Non-unique event_id makes the inbox blind. Publishing from application code and inserting the outbox is dual-write again.
You can delete outbox rows after the broker ack, or leave them and let topic compaction drop history. Pick one retention story and alert on PENDING age. Deleting immediately removes the audit trail you needed for replay.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Inbox (idempotent consumer)
inbox
msg_id TEXT PRIMARY KEY -- event_id or business id
status ENUM('PROCESSING','DONE','DUPLICATE','DEAD')
received_at TIMESTAMPTZ
processed_at TIMESTAMPTZ
attempts INT DEFAULT 0INSERT INTO inbox (msg_id, status) VALUES ($1, 'PROCESSING')
ON CONFLICT (msg_id) DO NOTHING
RETURNING msg_id- Row returned → you own processing. Run the handler.
UPDATE … DONE. - No row → someone already processed (or is processing). Skip.
If you crash after PROCESSING and before DONE, a lease / timeout (same idea as zombie idempotency keys) lets another worker finish — only if the handler is itself safe to retry (unique constraints on domain tables). Poison messages: increment attempts, after N move to DEAD / DLQ, do not block the partition forever.
Ordering
Global order does not scale. Per aggregate: (aggregate_id, sequence_id) monotonic. The consumer processes ascending sequence and pauses on a gap if your domain needs causal order (ledger). If the domain is commutative (cache invalidate), skip gap-wait.
Transactional guarantees
- Atomicity — order and outbox commit together or not at all.
- Durability — PENDING rows survive broker death.
- Isolation —
SKIP LOCKEDso pollers do not double-claim a live row. - Consistency model — write path exactly-once (DB); read path effectively-once (inbox).
Kafka EOS can wrap the poller’s produce + its offset if the poller is itself a Kafka consumer (CDC). It still does not write your orders table. Outbox remains the bridge.
Guarantees path — first hit and duplicate
Commit, publish, then the inbox fork. The first insert handles. A second insert skips.
- 1
POST /order
The client calls the service. The service does not produce inside this request. - 2
BEGIN
One database connection and one transaction for the order and the outbox. - 3
INSERT order
The domain row is staged in that transaction. - 4
INSERT outbox PENDING
Payload, topic, key, and sequence live on the outbox row in the same transaction. - 5
COMMIT
Order and outbox persist together. A crash before commit leaves neither. - 6
202 Accepted
Return before the broker ack. A PENDING row survives if the broker is down. - 7
SELECT PENDING SKIP LOCKED
The poller claims a live row. Another poller skips that lock. - 8
Publish
The poller sends to the broker. A produce failure leaves the row PENDING. - 9
ACK
Wait for the broker ack before treating the send as done. - 10
UPDATE SENT
Mark SENT only after the ack. A crash before this mark republishes. - 11
Deliver
The broker hands the message to the consumer. That work is a new transaction. - 12
INSERT ON CONFLICT DO NOTHING
The inbox insert is the dedupe gate. The row that returns owns the handler. - 13
First delivery: handle, then DONE
The insert wins. Run the handler and mark the inbox DONE. - 14
Duplicate delivery: skip
The insert affects 0 rows. Skip. At-least-once on the wire, effectively-once in the service.
Architecture choices
| Decision | Options | Prefer | Why |
|---|---|---|---|
| Outbox storage | Same DB vs other DB | Same DB | No 2PC |
| Propagation | CDC vs poller | CDC if you have it; poller always as fallback | Latency vs ops |
| Ordering | Global vs per-aggregate | Per-aggregate | Horizontal |
| Dedup | Memory vs inbox table | Inbox PK | Survives restart and multi-instance |
| Poison | Immediate DLQ vs N retries then DLQ | Exponential N then DLQ | Transient vs bug |
| Idempotency key | Random UUID vs business id | Business id + inbox PK | Downstream can dedupe too |
| Poller scale | One thread vs many + SKIP LOCKED | Many | Drain lag |
| Payload evolution | Add columns vs versioned JSON | Versioned envelope | Compatible consumers |
In-memory DB + broker (run this)
No Postgres, no Kafka. A tx flag: if we “crash” before commit, neither order nor outbox exists. If we commit then crash before publish, the poller still finds PENDING. Dual-write without an outbox loses the event. Inbox ON CONFLICT skips the second delivery.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Read the four plots:
- A crash before commit → empty DB; retry commits both.
- B crash after commit, before publish → poller recovers.
- C dual-write crash → order exists, no event, consume never runs.
- D poller send-then-crash → two broker messages, inbox size 1.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Interview Q&A
What is the dual-write problem?
Answer
Updating a database and publishing a message in two transactions. If either side fails, state and events diverge: lost updates or phantom events. There is no single COMMIT that covers Postgres and Kafka unless you build one (2PC) — you should not.
How does the outbox make the write exactly-once relative to domain state?
Answer
The outbox row is inserted in the same ACID transaction as the domain row. Callers observe the order only after both exist. If the process dies before COMMIT, neither exists and the client retries (with an HTTP idempotency key).
Crash after COMMIT, before the poller runs — is the event lost?
Answer
No. The outbox row is PENDING on disk. The poller or CDC will publish it. That is the recovery the dual-write path does not have.
Why can the poller still publish twice?
Answer
It may send, then crash before marking SENT. Another worker claims the still-PENDING row and sends again. The broker is at-least-once. The inbox (or an idempotent handler) makes processing effectively-once.
Poller or Debezium?
Answer
Debezium (WAL / outbox event router) for low latency and less hammering of the primary. Poller + SKIP LOCKED when you cannot run Connect, or as a fallback. Both need an inbox. CDC restarts also re-emit.
What does FOR UPDATE SKIP LOCKED do?
Answer
It lets many pollers SELECT PENDING rows without waiting on each other’s locks. Locked rows are skipped, not blocking. After a crash the lock releases and another worker takes the row.
Why not Kafka transactions instead of an outbox?
Answer
Kafka transactions atomic-commit records and offsets in Kafka. They do not include your INSERT INTO orders. Unless the business state lives only in Kafka (a stream processor with a transactional sink), you still have dual-write. Outbox is the pattern for “Postgres is the source of truth.”
What should msg_id be in the inbox?
Answer
A stable business id (order_id, event_id minted in the outbox insert). Do not mint a new UUID at consume time — every retry would look unique. Same rule as consumer dedup in the delivery-semantics lesson.
How do you handle poison outbox rows?
Answer
attempts++ on produce or handle failure. After N, FAILED or a dead-letter table, alert. Leaving them PENDING forever blocks that aggregate’s sequence if you require gap-free order.
Where does the HTTP Idempotency-Key fit?
Answer
The client key makes POST /orders retries safe at the API. The outbox makes the async emit safe. They stack: key claims the HTTP intent; the transaction writes order+outbox once; inbox claims the event. See keys.
When is table CDC the wrong tool and the outbox the right one?
Answer
Table CDC is a row mirror: search, cache, warehouse. The outbox is a domain fact (OrderPaid) that may cover several tables and must not track every OLTP column. One business transaction, one outbox insert, one public event. Capture of that insert is still CDC. The contract is not.
What does the Debezium outbox event router actually change?
Answer
It reads the committed outbox insert from the log and routes by aggregate type and event type, with aggregate_id as the key. It does not enlist Kafka in the database transaction. Offset flush is still at-least-once, so the inbox stays. Connector snapshot and slot behavior are the Debezium lesson.
The relay is down. Did the order commit fail?
Answer
No. The order and the outbox row committed together. The relay backlog is lag. Customers are not blocked on the bus. If the relay is a Debezium slot, also watch retained WAL so lag does not fill the primary disk.
Why not XA once you already run CDC?
Answer
CDC does not add a second commit for the business write. XA would try to commit Postgres and Kafka together and import coordinator failure. The outbox already made the intent durable. The consumer makes the effect once.
Can you delete outbox rows after publish?
Answer
Yes, after the broker ack, if you do not need them for audit or replay. CDC plus a compacted topic is the other retention plan. Deleting before ack, or having no retention story at all, loses the event or the ability to explain it. Say which one you operate.
Pitfalls
Whiteboard BEGIN → insert order → insert outbox → COMMIT → poller send → mark SENT → consumer inbox. Put an X on (1) before COMMIT, (2) after COMMIT before send, (3) after send before SENT. For each X, write what exists in orders, outbox, broker, inbox. Then repeat the three X’s on a dual-write diagram with no outbox — (2) is an unrecoverable miss.
Go Deeper
- microservices.io — Transactional outbox
- Debezium — Outbox event router
- Kafka — Exactly-once semantics
- Confluent — Transactional outbox pattern with Apache Kafka
- Debezium blog — Outbox pattern for microservices
- microservices.io — Idempotent consumer
- CDC cluster: Change Data Capture · Debezium and Kafka Connect · Exactly-once CDC pipelines · CDC failure modes
- Cluster: At-least-once vs exactly-once · Idempotency keys