Messaging
Part 5 of 5 · Kafka messagingSchema Evolution, Compatibility & Dead Letter Queues
Avro/Protobuf/JSON Schema; FORWARD/BACKWARD/FULL; poison→DLQ/retry; never block forever on bad payload.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Why registry plus bounded retry/DLQ wins
Prefer
Compatibility FULL + retry topic + DLQ
Producers register. Consumers fail closed to a delayed retry, then a DLQ with headers. The main partition keeps moving.
- Additive evolution with defaults keeps mixed-version fleets alive.
- Transient 5xx/timeouts retry; schema/validation poison does not loop forever.
- DLQ is an ops topic with alerts — not the hot path.
- Breaking redesigns get orders.v2 plus dual-write, not silent field reuse.
Alternative
JSON only, no schema, block on error
One bad payload holds the partition offset forever. Infinite retries page you for a poison pill. Field meaning changes without decode errors.
- FORWARD-only while consumers deploy slowly → old readers crash on new required fields.
- No DLQ → head-of-line blocking.
- Compatibility NONE on a shared topic is eventual chaos.
- Unmonitored DLQ is silent loss from the pipeline's point of view.
Register, consume, bound retries, page the DLQ
Vertical cards for phones. Same pipeline as the mermaid.
- 1
Produce with a compat check
Register/check the schema. Payload carries magic byte + schema id + bytes. - 2
Consume and decode
OK → business logic. Fail decode or validation → retry topic with delay, then commit the original offset. - 3
Bound the retries
Transient failures retry. Exhausted attempts go to the DLQ with headers. - 4
Ops on the DLQ
Alert, ticket, fix the schema or filter. Humans/tools drain DLQ — not the hot path.
Overview
Producers and consumers evolve independently. Schemas (Avro, Protobuf, JSON Schema) plus a Schema Registry enforce compatibility so old consumers can read new data (FORWARD), new consumers can read old data (BACKWARD), or both (FULL). Poison messages (fail decode / fail business validation forever) must not block a partition: route to a Dead Letter Queue (DLQ) topic after bounded retries (often via retry topics with delay). Never “block forever” on bad payloads in an at-least-once consumer.
You should be able to:
- Recite BACKWARD vs FORWARD in one sentence each.
- Draw produce → retry → DLQ without stalling offsets.
- Version a breaking field change as
orders.v2plus dual publish.
Architecture
1 Produce
- 1
Producer
- register / checkSchema Registry
- nextTopic
- 2
Schema Registry
- 3
Topic
- nextConsumer
2 Consume
- 4
Consumer
- decode OKBusiness logic
- decode or validate failRetry topic delayed
- 5
Business logic
- 6
Retry topic delayed
- attempt less than NConsumer
- exhaustedDLQ topic
Flow
- 7
DLQ topic
- nextPage / ticket
- 8
Page / ticket
- nextFix schema or poison filter
- 9
Fix schema or poison filter
Lesson map
Schema Evolution, Compatibility & Dead Letter Queues
Avro/Protobuf/JSON Schema; FORWARD/BACKWARD/FULL; poison→DLQ/retry; never block forever on bad payload.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB prod["Producer"] reg["Schema Registry"] top["orders"] cons["Consumer"] prod -->|register schema| reg prod -->|produce| top cons -->|fetch| top cons -->|commit offset do| top
Formats and compatibility
| Format | Strengths | Weaknesses | Typical use |
|---|---|---|---|
| Avro + Registry | Compact, rich compat rules | JVM-centric tooling bias | Confluent ecosystems |
| Protobuf | Strong codegen, gRPC kinship | Schema story varies by registry | Polyglot services |
| JSON Schema | Human readable | Verbose; weaker defaults | Public events / flexibility |
| JSON only, no schema | Fast start | Silent breakages | Prototypes only |
| Compat mode | Writer/reader guarantee | Breaks when |
|---|---|---|
| BACKWARD | New code reads old data | Removing required fields without defaults |
| FORWARD | Old code reads new data | Adding required fields without defaults |
| FULL | Both | Either direction breaks |
| NONE | Chaos | Always eventually |
What fails if you choose wrong
- FORWARD-only while consumers deploy slowly → old readers crash on new required fields.
- No DLQ → one bad message stalls partition offset forever (head-of-line blocking). That looks like consumer lag.
- Infinite retries on poison → lag + cost + pager fatigue.
- Changing field meaning in place (“reuse status int”) → semantic corruption without decode errors.
Evolution rules and DLQ design
Evolution rules of thumb (Avro-ish)
- Add optional field with default → usually BACKWARD-safe.
- Delete a field consumers still need → breaks BACKWARD.
- Rename via alias; do not repurpose ordinals lightly.
- Prefer additive evolution; version topics (
orders.v2) for breaking changes. Dual-write during cutover pairs with outbox.
DLQ design
- Include headers:
original_topic,partition,offset,exception,attempt,correlation_id. - Separate retry (transient: 5xx, timeouts) from DLQ (permanent: schema, validation).
- Consumers of DLQ are humans/tools — not the hot path.
- After publishing to retry/DLQ, commit the original offset so ALO does not stall. Side effects still need idempotency if you later replay from DLQ.
Sequence
- 1
Producer
1 Compat check on register
- 2
Producer → Schema Registry
register schema FULL
- 3
Producer → orders
produce magic-byte plus schema-id plus payload
- 4
Consumer
2 Fail closed to retry/DLQ
- 5
Consumer → orders
fetch
- 6
Consumer → Consumer
decode fails
- 7
Consumer → orders.retry.1
publish with attempt=1
- 8
Consumer → orders
commit offset do not block
- 9
orders.retry.1
3 Exhaust then DLQ
- 10
orders.retry.1 → Consumer
redelivery
- 11
Consumer → orders.dlq
attempt=N exhausted
Compatibility and DLQ (run this)
v2 adds currency with default — BACKWARD-safe. A payload missing required amount exhausts retries into the DLQ.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Pros / cons
| Pattern | Pros | Cons | Prefer when |
|---|---|---|---|
| Registry + FULL | Safe evolution | Process overhead | Many teams on one topic |
| Topic versioning | Clean breaks | Dual-write cost | Breaking redesigns |
| Retry + DLQ | Protects lag | Ops to drain DLQ | All at-least-once consumers |
| Block on error | None | Partition stall | Never in prod |
Interview Q&A
BACKWARD vs FORWARD?
Answer
BACKWARD: new consumer reads old messages. FORWARD: old consumer reads new messages. FULL is both.
Why DLQ?
Answer
Avoid head-of-line blocking on poison payloads while preserving the bad record for forensics.
Retry vs DLQ?
Answer
Retry = transient failures. DLQ = permanent / exhausted attempts.
What travels with a DLQ record?
Answer
Original topic/partition/offset, error, attempt count, headers/correlation ids.
Is JSON without a registry OK?
Answer
Only for prototypes. Production needs explicit compatibility.
How do you break a schema safely?
Answer
New topic version + migrate consumers; dual publish during cutover.
Pitfalls
v1 has id + amount. A producer ships v2 with required currency and no default. Old consumers are still out. Name the compat mode that breaks, the user-visible failure, and the fix (default vs orders.v2). Then a payload fails Avro decode: write retry vs DLQ vs block, and whether you commit the original offset.