Data engineering
Part 3 of 5 · CDC & DebeziumDebezium & Kafka Connect — Snapshots, Offsets, Schema History & Heartbeats
Debezium snapshots a consistent read, then streams from a stored position. Connect offsets are the resume token. Schema history decodes DDL. Heartbeats advance a quiet slot so WAL can be released. Domain events still go through the outbox lesson.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Debezium versus a homegrown decoder
Prefer
Debezium on Kafka Connect
A snapshot protocol, offset topic, schema history, and SMTs already exist. You still operate slots, history topics, and lag.
- Initial, when-needed, never, and schema-only snapshot modes.
- Outbox event router is an SMT, not a second database commit.
- Wide database coverage if you already run Connect.
Alternative
pg_recvlogical and a custom publisher
Full control, and you own snapshot consistency, DDL, retries, and the resume token. Worth it only when a platform team is the product.
- Easy to gap the snapshot-to-stream handoff.
- Easy to lose the DDL needed to decode old events.
- Connect’s opaque failures become your opaque failures.
Overview
Debezium is the usual CDC connector family. The interview is not “what is a Kafka topic.” It is snapshot versus streaming, where offsets live, why a schema history topic exists, and what a heartbeat is for.
Connect stores connector offsets. Debezium also writes schema history so it can decode a table that has changed shape since the connector started. Data topics carry the change events. Sinks and consumers sit after that. Broker internals stay in the messaging cluster.
Flow
- 1
1. OLTP database
- next2. Slot or binlog position
- 2
2. Slot or binlog position
- next3. Debezium connector task
- 3
3. Debezium connector task
- next4. Schema history topic
- next5. Connect offsets topic
- next6. Change data topics
- 4
4. Schema history topic
- 5
5. Connect offsets topic
- 6
6. Change data topics
- next7. Sinks and consumers
- 7
7. Sinks and consumers
Lesson map
Debezium & Kafka Connect — Snapshots, Offsets, Schema History & Heartbeats
Debezium snapshots a consistent read, then streams from a stored position. Connect offsets are the resume token. Schema history decodes DDL. Heartbeats advance a quiet slot so WAL can be released. Domain events still go through the outbox lesson.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB db["1. OLTP database"] slot["2. Slot or binlog position"] conn["3. Debezium connector task"] hist["4. Schema history topic"] db -->|1. OLTP database to 2. Slot or binlog position| slot slot -->|2. Slot or binlog position| conn conn -->|3. Debezium connector task| hist
Snapshot, then stream
Snapshot mode names vary by version and connector. The ones you should be able to say:
- initial — snapshot, then stream. The usual start.
- when_needed / incremental variants — snapshot if the offset is missing or you ask for a repair.
- never — stream only. You already loaded the data, or you accept a gap.
- schema_only — capture structure, do not emit the current rows.
- ad hoc / incremental — signal table or incremental snapshot so you do not lock the world to add one table.
Typical initial sequence:
- Take a consistent read of the included tables.
- Emit those rows as
op=r(read) events. - Record the streaming start position (LSN, GTID, or binlog file and position).
- Release the snapshot lock or export.
- Stream
c,u, anddfrom that position.
The goal is no gap. A duplicate at the boundary is acceptable when sinks are idempotent. A missing row is not. Idempotency is the next series page, after you have read the outbox lesson if the payload is a domain event.
Flow
- 1
1. Begin a consistent snapshot
- next2. Emit snapshot rows as op r
- 2
2. Emit snapshot rows as op r
- next3. Persist start LSN or GTID
- 3
3. Persist start LSN or GTID
- next4. End the snapshot
- 4
4. End the snapshot
- next5. Stream from that position
- 5
5. Stream from that position
- next6. Emit create update delete
- 6
6. Emit create update delete
- next7. Flush offsets on a cadence
- 7
7. Flush offsets on a cadence
Offsets
A Connect offset is the connector’s resume token. On Postgres that is an LSN plus transaction pieces. On MySQL it is a binlog file and position, or a GTID.
Flush cadence is not once per event. A crash replays from the last flushed offset. That is at-least-once capture. Do not “fix” a lagging offset by hand unless you also have a replay or backfill plan. Rewind into a non-idempotent sink is the failure-modes lesson.
Workers run tasks. A rebalance or a stuck task shows up as lag. Watch both the task and the slot. The rebalance protocol itself belongs to the Kafka messaging cluster, not here.
Schema history
DDL (add column, change type) is appended to a schema history topic. The connector uses that history to decode events written before and after the change.
Downstream still needs a contract. Prefer additive columns. Do not reuse or rename a field without a migration window. Raw Debezium envelopes are wide; many teams project them into domain events. If the event was never the row, do that projection in the database via the outbox, not by hoping consumers ignore columns.
Deleting the history topic to “clean up” makes the connector unable to decode. Treat history like a stateful system, with retention and backups, not like a debug log.
Compatibility types (backward, forward, full) are taught in Schema Evolution, Compatibility & Dead Letter Queues. Use that page for registry rules. Use this page for why the connector itself must remember DDL.
Heartbeats
On a low-traffic database the confirmed LSN can stall even while other activity, or no activity, keeps WAL around for the slot. Heartbeat queries or heartbeat events advance the offset on a timer so the slot can release WAL.
Interview line: a heartbeat is an operations safety valve for slot retention. It is not a business event. If the connector process is dead, heartbeats stop too. You still need an alert on retained WAL bytes.
Configuration themes
Remember the concern, not every property name.
| Concern | What to reason about |
|---|---|
| Include and exclude | Blast radius, PII, cost |
| Tombstones on delete | Compacted topics need a null value to drop a key |
| Topic routing | One topic per table, an SMT reroute, or the outbox SMT |
| Snapshot parallelism | Large tables want chunked or incremental snapshots |
| Signal tables | Ad hoc snapshot without a full connector restart |
Adding a table is an include-list change plus, usually, an incremental snapshot. Watch the history topic and the primary’s IO. Do not snapshot a huge table in one shot at peak.
The outbox sits between this page and exactly-once
Table capture is what this lesson’s op=r/c/u/d stream is for. When the product needs OrderPaid rather than “column status changed,” the application inserts an outbox row in the same transaction and the Outbox Event Router SMT routes it. That pattern, the poller alternative, and the inbox are Transactional Outbox & Inbox Patterns.
Series order jumps from this page to Exactly-Once CDC Pipelines because the outbox page already belongs to Idempotency. Read it before you design sink keys. Exactly-once effects do not care whether the envelope came from a table topic or an outbox topic, but the identity you dedupe on does.
Offset resume (run this)
Flush every third LSN. Crash at LSN 5, before the next flush. Resume redelivers the events after last_flushed. The sink must tolerate that.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Interview Q&A
What does a Debezium snapshot do?
Answer
A consistent initial read of the selected tables, emitted as read events, then streaming from the position recorded at the end of that read. The point is no gap. Duplicates at the edge are a sink problem.
Where are offsets stored?
Answer
In the Kafka Connect offsets topic for a Kafka Connect deployment. The payload is the resume token: LSN, GTID, or binlog coordinates. A file offset store is a different deployment; the meaning of the token is the same.
Why does schema history exist?
Answer
So the connector can decode events across DDL for the life of the capture. Lose the history topic and you often cannot decode, which means a snapshot rebuild rather than a clever edit.
Why heartbeats?
Answer
To advance offsets when the captured tables are quiet, so the replication slot can release WAL. They are not business events. A dead connector does not heartbeat.
Are snapshot and streaming duplicates a bug?
Answer
They are a boundary you should expect. Design the consumer to apply op=r and later changes idempotently. That is the exactly-once lesson.
What is an SMT?
Answer
A single message transform inside Connect. The outbox event router is the one that turns an outbox row into a routed domain event. The transaction that created the row is explained in Transactional Outbox and Inbox Patterns, not by the SMT.
How do you add a table?
Answer
Update the include list and trigger an incremental or ad hoc snapshot. Watch history, slot lag, and primary IO. A full snapshot of every table is the expensive version.
Connector task versus Connect worker?
Answer
Workers host tasks. Task failure, restarts, and rebalances move lag. Monitor task state and source lag, not only “the cluster is up.”
What should you monitor on a quiet primary?
Answer
Retained WAL for the slot, heartbeat success, and offset movement. Quiet business traffic plus a stuck slot is how disks fill with no user-visible writes.
Pitfalls
A Postgres primary pages on disk. Traffic to the captured tables has been low for two days. List the three checks in order: slot retained bytes, whether the task is running, whether heartbeats are advancing the offset. Then say what you will not do (delete history, skip the LSN).
Go Deeper
- Debezium reference documentation
- Debezium connector for PostgreSQL
- Kafka Connect — offset storage
- Debezium — Outbox event router
- Between this page and exactly-once: Transactional Outbox & Inbox Patterns
- Next in series: Exactly-Once CDC Pipelines