Stop Reprocessing Everything: Build Incremental Data Pipelines on AWSHow to use AWS Glue, Amazon S3, and PySpark to process only new and changed data without making your pipeline fragile. Imagine you have a data pipeline that processes 100 GB of data every morning. TheOct 16 · 12 min readrahul-dataeng.hashnode.dev
When Delta Lake Jobs Collide: Troubleshooting Concurrency Issues in AWS GlueHave you ever run two AWS Glue jobs against the same Delta table and noticed that one job fails while the other completes successfully? The confusing part is that both jobs might work perfectly when eOct 12 · 10 min readrahul-dataeng.hashnode.dev
AWS Step Functions vs Apache Airflow: Similar Purpose, Different Use Cases When I started exploring data engineering workflows, one question that came to mind was: if AWS Step Functions and Apache Airflow can both orchestrate workflows, why do we need both? At first glance, Oct 11 · 7 min readrahul-dataeng.hashnode.dev
Stop Hardcoding API Keys: A Practical Guide to Fetching Secrets from HashiCorp VaultHave you ever stored a database password or API key directly in your application code just to get something working? I think many developers have done something similar while learning or building a quOct 9 · 8 min readrahul-dataeng.hashnode.dev
When AWS Glue Couldn’t Read My S3 Files: Solving the Small Files ProblemIf you are working with AWS Glue and Amazon S3, you may eventually come across a problem that doesn't look serious at first: There are too many small files in S3. A file that is only a few KB doesn'1RSep 26 · 13 min readrahul-dataeng.hashnode.dev
