Step-by-Step Guide to Setting Up Medallion Architecture on AWSModern analytics platforms require structured and reliable data processing pipelines. The Medallion Architecture—Bronze → Silver → Gold—provides a standardized way to achieve this while maintaining quality, lineage, and re-processing capabilities. In...Dec 7, 2025 · 3 min readthedatapipeline.hashnode.dev
Efficiently Load Millions of Rows Daily from S3 into AWS RedshiftA few months ago I built a pipeline for a logistics analytics team that collects package events—delivery scans, route status updates, warehouse entries, etc. The events come from 11 distributed warehouses across India, aggregating to ~40M records/day...Dec 6, 2025 · 4 min readthedatapipeline.hashnode.dev
Must-Know AWS Glue Interview Questions and AnswersWhat is AWS Glue Crawlers? A Glue crawler is simply a service that scans your data source—mostly S3 in data lake setups—and automatically figures out the schema and creates tables inside the Glue Data Catalog. It can detect new partitions and even up...Dec 5, 2025 · 6 min readthedatapipeline.hashnode.dev
Top Snowflake Interview Questions for Data Engineers1. Explain Snowflake Architecture? Snowflake uses a fully decoupled architecture consisting of three independent layers: Storage, Compute, and Cloud Services. The Storage layer holds all data in compressed, columnar structures, internally broken into...1BDec 4, 2025 · 4 min readthedatapipeline.hashnode.dev
How to Implement Stream-Based SCD Type 2 in SnowflakeSlowly Changing Dimensions (SCD) Type-2 is the go-to pattern when you need full historical tracking of your dimensional tables. But running a naïve column-by-column comparison becomes messy fast. A much cleaner way? 👉 Create an MD5 hash of all busin...Dec 3, 2025 · 2 min readthedatapipeline.hashnode.dev
