PPipeline Whiteboard11h ago · 10 min readCDC Deletes, Tombstones and the Idempotent MERGEOrder 1042 was deleted at 14:10. At 15:00 it's back in the warehouse, marked shipped. Nobody re-created it. A retried batch did. This is Part 4 of CDC in Production. In Part 3 we caught silent data lopipelinewhiteboard.hashnode.dev
TThyne Ventures1d ago · 2 min readIs your Snowflake warehouse queueing? One read-only query to checkSlow Snowflake dashboards aren't always slow queries. Often the query runs in two seconds but waits twenty in line because the warehouse is full. Here's a read-only query that shows which warehouses q1thyneventures.hashnode.dev
BBalaji Kottana4d ago · 10 min readFrom On-Prem to a Modern Data Platform, Part 1: The Architecture End to EndMost migration write-ups stop at the target diagram: source, cloud storage, warehouse, dashboard. The diagram is the easy part. What decides whether a platform holds up in production is everything arodatanarratives.hashnode.dev
TThyne Ventures3d ago · 2 min readIs your Snowflake query reading the whole table? Check partition pruningPruning is the biggest performance lever in Snowflake. A query that scans 2% of a table's micro-partitions instead of 100% is often 10-50x faster on the same warehouse. This read-only query finds the thyneventures.hashnode.dev
TThyne Ventures4d ago · 2 min readFind the Snowflake queries spilling to remote storage (one read-only query)When a Snowflake query's intermediate results don't fit in warehouse memory, it spills: first to the warehouse's local SSD, then to remote cloud storage. Remote spill often makes a query 5-10x slower,thyneventures.hashnode.dev
TThyne Ventures4d ago · 5 min readWhy is this Snowflake query slow? 3 ACCOUNT_USAGE queries that tell you"Just make the warehouse bigger" is the most expensive way to tune Snowflake, and often it doesn't even work. A bigger warehouse speeds up the execution phase of a query. It does nothing for a query tthyneventures.hashnode.dev
PPipeline WhiteboardOct 23 · 9 min readChange Data Capture (CDC) Explained: From the Transaction Log to a Snowflake MERGEYour sales dashboard says there are 500 units in stock. Run a query against production and you get 12. Nobody broke anything. The nightly batch job ran nine hours ago, and in those nine hours the prodpipelinewhiteboard.hashnode.dev
AAriftlyOct 20 · 5 min readOracle to Snowflake migration validation: a six-gate acceptance frameworkOriginally published by Ariftly. This version focuses on the engineering method. The test scenarios below are synthetic specifications, not executed benchmark results. A successful SQL conversion ansariftly.hashnode.dev
AAira FrancoOct 6 · 11 min readLearning Data Orchestration with DagsterUp to this point in my data engineering project, I had already worked with the individual parts of a modern data stack: dlt for ingestion, Snowflake for storage, dbt for transformation, and dbt tests dataengineeringlearning.hashnode.dev
AAira FrancoOct 5 · 20 min readInside a Modern Data Warehouse Lifecycle: Snowflake, dlt, dbt, and Data ModelingThis blog walks through the full lifecycle of a modern data warehouse, from setting up Snowflake infrastructure and access control to ingesting API data with dlt, transforming it with dbt, modeling it1dataengineeringlearning.hashnode.dev
