Pipeline Whiteboard (@pipelinewhiteboard)

CDC Deletes, Tombstones and the Idempotent MERGE

Order 1042 was deleted at 14:10. At 15:00 it's back in the warehouse, marked shipped. Nobody re-created it. A retried batch did. This is Part 4 of CDC in Production. In Part 3 we caught silent data lo

19h ago · 10 min readpipelinewhiteboard.hashnode.dev

The Silent CDC Bug: Data Loss With No Errors

Every dashboard is green. The connector says RUNNING, capture lag is under a second, and the warehouse is missing 4,812 orders. Nobody gets paged, because nothing failed. This is Part 3 of CDC in Prod

3d ago · 9 min readpipelinewhiteboard.hashnode.dev

Debezium Postgres to Kafka: Every Setting Explained

About fifteen lines of JSON. That's all it takes to stream every insert, update and delete from Postgres into Kafka with Debezium. And two of the defaults hiding in those lines will hurt you in produc

3d ago · 10 min readpipelinewhiteboard.hashnode.dev

Change Data Capture (CDC) Explained: From the Transaction Log to a Snowflake MERGE

Your sales dashboard says there are 500 units in stock. Run a query against production and you get 12. Nobody broke anything. The nightly batch job ran nine hours ago, and in those nine hours the prod

Oct 23 · 9 min readpipelinewhiteboard.hashnode.dev

Pipeline Whiteboard

About

Available for

Pipeline Whiteboard's blogs

CDC Deletes, Tombstones and the Idempotent MERGE

The Silent CDC Bug: Data Loss With No Errors

Debezium Postgres to Kafka: Every Setting Explained

Change Data Capture (CDC) Explained: From the Transaction Log to a Snowflake MERGE

Pipeline Whiteboard

About

Available for

Pipeline Whiteboard's blogs

CDC Deletes, Tombstones and the Idempotent MERGE

The Silent CDC Bug: Data Loss With No Errors

Debezium Postgres to Kafka: Every Setting Explained

Change Data Capture (CDC) Explained: From the Transaction Log to a Snowflake MERGE

Pipeline Whiteboard

About

Available for

Pipeline Whiteboard's blogs

CDC Deletes, Tombstones and the Idempotent MERGE

The Silent CDC Bug: Data Loss With No Errors

Debezium Postgres to Kafka: Every Setting Explained

Change Data Capture (CDC) Explained: From the Transaction Log to a Snowflake MERGE

Search Hashnode

Pipeline Whiteboard

About

Available for

Pipeline Whiteboard's blogs

CDC Deletes, Tombstones and the Idempotent MERGE

The Silent CDC Bug: Data Loss With No Errors

Debezium Postgres to Kafka: Every Setting Explained

Change Data Capture (CDC) Explained: From the Transaction Log to a Snowflake MERGE