Building a Metadata-Driven Dagster Platform: One Generic Job, N PipelinesMost Dagster tutorials end at “define an asset, attach a schedule.” This one starts where that ends: how do you run an unbounded, user-defined set of pipelines on a fixed codebase, with no redeploys pOct 18 · 5 min readsafi-io.hashnode.dev
End-to-End Secrets Management for the Modern Data PlatformIntroduction Over the past few months, I have been heavily involved in building modern data platforms. From orchestrating pipelines and managing distributed services to deploying containers and handliMay 23 · 5 min readsafi-io.hashnode.dev
Encapsulating External Dependencies with Dagster ResourcesWhy Encapsulation Matters in Data Pipelines In real-world pipelines, external systems (databases, APIs, queues) are stateful, unreliable, and slow. Directly calls to them inside ops/assets creates: TApr 12 · 5 min readsafi-io.hashnode.dev
Mastering Dagster SensorsIntroduction What are Dagster Sensors? Dagster Sensors are components that monitor external systems (like storage, APIs, or databases) and trigger pipeline runs when specific conditions are met. They Apr 6 · 5 min readsafi-io.hashnode.dev
How Docker Breaks UFW and Alters iptablesLet me set the context for this article: I was deploying some applications on my VPS, and while I had configured UFW (Uncomplicated Firewall), the ports published by Docker were still exposed directly to the internet. In this article, we’ll explore t...Jan 15 · 4 min readsafi-io.hashnode.dev
