CDC & Debezium
Studies in this cluster, in series order. Each one keeps its own URL.
Data engineering
Pipelines, sketches, approximate aggregations, object storage, and stream processing with event time, windows, and exactly-once sinks.
CDC & Debezium
5 studies- 1.Change Data Capture — WAL Tailing, Debezium & Event PipelinesDual-write splits one business fact across a database commit and a later publish. CDC reads the database change log so the commit is the event. This hub maps log versus poll, Debezium, the existing outbox lesson, exactly-once effects, and failure modes.
- 2.WAL Tailing vs Query-Based CDC — Log vs Poll TradeoffsPoll CDC reads a watermark column and misses hard deletes. Log CDC reads commit order from WAL or binlog and keeps a replication slot until the consumer confirms. Pick the log when you can operate it.
- 3.Debezium & Kafka Connect — Snapshots, Offsets, Schema History & HeartbeatsDebezium snapshots a consistent read, then streams from a stored position. Connect offsets are the resume token. Schema history decodes DDL. Heartbeats advance a quiet slot so WAL can be released. Domain events still go through the outbox lesson.
- 4.Exactly-Once CDC Pipelines — Idempotent Consumers, Keys & At-Least-Once RealityCDC capture is at-least-once. Exactly-once effects come from a stable key plus an idempotent sink: an LSN guard for projections, an inbox for side effects. Kafka transactions do not make an email or a charge exactly-once.
- 5.CDC Failure Modes — Lag, Schema Breaks, Tombstones, Backfills & ReplaysCDC fails in production as slot disk, breaking DDL, missing tombstones, and a panicked offset rewind. Freshness is an SLO. Schema changes expand then contract. Rebuild a projection beside the live one, then swap.