Messaging
Part 3 of 5 · Kafka messagingDelivery Semantics — At-Least-Once, At-Most-Once & Exactly-Once
AMO vs ALO vs EOS (idempotent producer+transactions); external side effects still need idempotency keys.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Default: at-least-once plus application idempotency
Prefer
ALO + unique event_id + outbox/inbox
Retries and commit-after-process duplicate the wire. A unique constraint or inbox makes the business look once. Kafka EOS is optional inside the log.
- Works with any at-least-once channel, not just Kafka transactions.
- Outbox commits domain row + event in one DB transaction.
- HTTP/Stripe still uses Idempotency-Key — Kafka cannot roll back Stripe.
- Poison payloads go to a DLQ instead of stalling the partition.
Alternative
Believe the EOS checkbox covers the world
Idempotent producer + transactions are exactly-once inside Kafka. A non-transactional sink or HTTP call is still at-least-once.
- At-most-once for payments vanishes charges.
- ALO without upsert double-charges and double-emails.
- Shared transactional.id across pods → zombie producers.
- read_committed off → consumers see aborted tx records.
Pick the tolerance, then implement it
Vertical cards for phones. Same branches as the mermaid.
- 1
Ask which is worse: loss or duplicate
Metrics can lose. Ledgers cannot. Most business events prefer duplicates if the handler is idempotent. - 2
At-most-once
acks=0/1 or commit offset before work. Simple. Loss on crash. - 3
At-least-once
Retries + acks=all. Process, then commit. Duplicates happen. Add an idempotency store keyed by event_id. - 4
Kafka EOS
Idempotent producer + transactional.id. beginTxn, write outputs, sendOffsetsToTransaction, commitTxn. isolation read_committed. - 5
Outside Kafka
DB unique key, outbox/inbox, or HTTP Idempotency-Key. Transactions do not undo Stripe.
Overview
At-most-once: lose messages rather than duplicate (commit/ack before processing, or fire-and-forget). At-least-once: process then commit — survivors under retries; duplicates happen. Exactly-once (EOS): Kafka’s idempotent producer + transactions (read-process-write) give effectively once within the Kafka transactional boundary; side effects outside Kafka still need idempotency keys.
Interviewers expect “exactly-once is a protocol + application design,” not a magic checkbox. Brokers, ISR, and groups: the hub. Triangle and effectively-once default: at-least-once vs exactly-once.
You should be able to:
- Fill the producer/consumer columns for AMO, ALO, EOS.
- Crash after side effect, before offset commit — name the duplicate.
- Refuse to put Stripe inside
commitTransaction.
Decisions
- ?
1 Lose or duplicate worse?
- lose OKAt-most-once
- dup OK if idempotentAt-least-once plus idempotency
- Kafka-internal EOSIdempotent producer plus txn
- 2
At-most-once
- nextcommit offset before process
- 3
At-least-once plus idempotency
- nextprocess then commit
- 4
Idempotent producer plus txn
- nextbeginTxn / sendOffsets / commitTxn
- 5
commit offset before process
- 6
process then commit
- nextIdempotency store keyed by event_id
- 7
beginTxn / sendOffsets / commitTxn
- nextIdempotency store keyed by event_id
- 8
Idempotency store keyed by event_id
Lesson map
Delivery Semantics — At-Least-Once, At-Most-Once & Exactly-Once
AMO vs ALO vs EOS (idempotent producer+transactions); external side effects still need idempotency keys.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB c["Consumer"] p["Txn producer"] t["Input topic"] o["Output topic"] c -->|fetch| t c -->|process| p p -->|produce results| o
Comparative table
| Semantic | Producer | Consumer | Duplicates | Loss | Complexity |
|---|---|---|---|---|---|
| At-most-once | acks=0/1, no retry care | Commit before work | Rare | Possible | Low |
| At-least-once | Retries + acks=all | Work then commit | Yes | Rare if durable | Medium |
| Exactly-once | Idempotent + transactional.id | read_committed + txn commit | Suppressed in-txn | Rare | High |
What fails if you choose wrong
- At-most-once for payments → vanished charges.
- At-least-once without idempotent upsert → double charges / duplicate emails.
- EOS without fencing / unique
transactional.id→ zombie producers corrupt streams. - Believing EOS covers HTTP calls to Stripe → still need idempotency keys.
Idempotent producer, transactions, application keys
Idempotent producer
enable.idempotence=true assigns producer PID + sequence numbers per partition so the broker dedupes retries. Needed foundation for EOS; alone it does not make consume-transform-produce atomic.
Transactions (EOS read-process-write)
- Consumer reads (
isolation.level=read_committed). - Producer begins a transaction, writes outputs.
sendOffsetsToTransactionfor the input offsets.commitTransaction— outputs and offsets commit atomically; abort → neither visible.
Application idempotency
Store event_id (or hash of a natural key) in DB with a unique constraint; treat duplicate insert as success. Outbox for DB→Kafka: transactional outbox & inbox. Crash-after-process-before-commit is why rebalances redeliver.
Sequence
- 1
Consumer
1 Poll committed records
- 2
Consumer → Input topic
fetch read_committed
- 3
Consumer
2 Transactional write plus offsets
- 4
Consumer → Txn producer
process
- 5
Txn producer → Txn coordinator
beginTxn
- 6
Txn producer → Output topic
produce results
- 7
Txn producer → Txn coordinator
sendOffsetsToTransaction
- 8
Txn producer → Txn coordinator
commitTxn
- 9
Output topic
3 Downstream sees atomic commit
Semantics simulator (run this)
Crash timing on a side-effect list. AMO loses. ALO duplicates. App-level unique event_id suppresses the retry.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Pros / cons
| Approach | Pros | Cons | Prefer when |
|---|---|---|---|
| At-most-once | Simple, low latency | Loss | Metrics, best-effort |
| At-least-once + idempotency | Practical default | Need unique keys | Most business events |
| Kafka EOS txn | Atomic multi-topic | Ops + latency | Stream processors / Kafka Streams |
Interview Q&A
Default semantic most teams get?
Answer
At-least-once (retries + commit after process). That is not exactly-once until the handler is idempotent.
Does idempotent producer equal exactly-once end-to-end?
Answer
No — only dedupes producer retries to a partition. Consumer redelivery and external sinks are separate.
What is read_committed?
Answer
Consumers skip uncommitted (and aborted) transactional records. Without it, EOS readers can see phantoms.
How do you EOS with a DB side effect?
Answer
Transactional outbox or an inbox table with unique event_id, not Kafka txn alone. Kafka EOS does not include your SQL session.
Zombie fencing?
Answer
Unique transactional.id + epoch so old producer instances are fenced. Do not reuse the same id across scaled pods.
Why duplicates with at-least-once?
Answer
Crash after side effect, before offset commit → redelivery. Same story as HTTP retries without idempotency keys.
Pitfalls
Draw produce → Kafka → consumer → Stripe. Mark AMO (commit first), ALO (process first), EOS txn (Kafka only), and ALO + Idempotency-Key + outbox. Crash after Stripe 200, before offset commit. Write loss, double charge, or replay no-op in each column.
Go Deeper
- Kafka — Exactly-once semantics
- Confluent — Exactly-once semantics are possible
- Confluent — Transactions in Apache Kafka
- Kleppmann — Making Sense of Stream Processing
- Kleppmann — Idempotent consumers (YouTube)
- Sibling lesson: At-least-once vs exactly-once
- Next: Consumer rebalancing, lag & backpressure