Discussion on "Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)"

Using Nessie branches together with dbt is a strong pattern here: run dbt against a branch, run tests there, and only merge into main when they pass, so Trino consumers never see half-built tables. On a homelab, also schedule Iceberg maintenance (expire_snapshots, remove_orphan_files and rewrite_data_files) early, because frequent small dbt runs pile up snapshots and tiny files on Ceph fast, and Trino query times degrade before you notice why.

Thanks. That's how dbt runs here: write-audit-publish on a dbt-audit branch, which merges to main only when the whole build passes, so Trino never sees a half-built table.

On maintenance, Nessie changes the picture a bit. Iceberg tables behind Nessie keep a single snapshot in their own metadata, because the history lives in Nessie's commit log. So expire_snapshots has nothing to do. Orphan and unreferenced files are handled by Nessie GC, which runs weekly with a 30-day cutoff (still in record-only mode while I build trust in it).

rewrite_data_files is the real gap, and you're right about the cause. Compaction isn't scheduled yet, and the table written every minute is the one piling up small files on Ceph. That's next on the list.

This is the kind of homelab project that makes you realize how many moving parts are hiding behind a “simple” data platform. I really liked that this isn’t just a list of tools thrown onto Kubernetes—the decisions seem to come from actually running into problems and then simplifying where possible.

The choice to use Ceph, Iceberg, Nessie, Trino, dbt, MLflow and Kubeflow as pieces of one workflow is especially interesting. The “Git for data” approach with Nessie and the emphasis on reproducibility make a lot of sense for ML, where knowing exactly what data produced a model can be just as important as tracking the code.

I also appreciated the honesty around the parts that are not perfect yet. Keeping some bundled databases in Kubeflow, deferring streaming, and having Spark installed but not forcing it into the daily pipeline feels much more realistic than pretending everything needs to be production-ready from day one.

The troubleshooting details around Istio, Vault, oauth2-proxy and Keycloak are probably the most valuable part for anyone attempting something similar. Those are exactly the kinds of issues that don't show up in architecture diagrams but can consume hours in a real deployment.

Really impressive build—and even better that you're documenting the traps along the way, not just the final architecture.

Thanks, I appreciate it. The traps sections are the ones I'd have wanted when I started: most of the hours went into auth chains and storage settings, not the parts the architecture diagram shows. More of that in the next articles in the series.

The "one source of truth per concern" principle is what makes this tractable — refusing stateful chart defaults is where most homelab attempts fall apart. The Nessie write-audit-publish workflow reads like Git semantics for data, which is what makes reproducible training believable rather than aspirational. How do you size the ops overhead of keeping this whole stack healthy on a homelab node — is it a weekend project that mostly runs itself, or a constant maintenance tax compared to paying for managed?

Honestly, it mostly runs itself day to day. The lab is powered off every night and cold-boots every morning, and everything comes back on its own: ArgoCD syncs from Git, CloudNativePG brings Postgres back, and the daily pipeline runs on schedule. That nightly restart is a good test that nothing depends on hand-applied state.

The tax is upgrades, not operations. Moving Kubeflow, Istio, Knative or Ceph one version means checking how each fits with the others first, and that's where the hours go. Managed services would remove most of that. I'd still choose the homelab, because the point is to learn where the parts connect, and you only learn that by owning them.

The dbt + Apache Iceberg + Project Nessie workflow. When a dbt build fails halfway through, how do you ensure that temporary Iceberg files and snapshots created on the dbt-audit branch are cleaned up safely without affecting tables referenced by existing Nessie tags or MLflow models? Would you recommend Nessie garbage collection with a retention policy, or is there another approach for handling this in production?

Good question. Two things keep it safe here: the failed build never touches main, and nothing deletes files directly.

dbt runs write-audit-publish. Each run first points the dbt-audit branch at main's head, runs dbt build against that branch, and merges to main only if the whole build passes. If it fails halfway, main is untouched, and whatever it wrote stays on dbt-audit so I can inspect it. The next run resets dbt-audit to main again, and those half-written commits stop being reachable from any branch or tag.

Cleanup is Nessie GC, not anything dbt-side. It marks as live every file reachable from any branch or tag within the cutoff (30 days by default), and sweeps the rest. Tags are references too, so a train-football- tag keeps its snapshot's files alive for as long as the tag exists. I give those tags a cutoff of 1 so GC only keeps the commit the tag points at, not 30 days of history behind it.

What decides how long a tag lives is a separate retention job. It keeps a tag if a Model Registry version names it as its training data, if it's under 30 days old, or if it's the newest one. It deletes the rest, and only at the hash it listed (Nessie rejects the delete if the tag moved). It fails closed: if the registry can't be read, it deletes nothing. One caveat: it checks the Kubeflow Model Registry, not MLflow, so an MLflow run that records a tag doesn't protect it on its own.

So yes, I'd use Nessie GC with retention, but put the retention policy on your references (which tags exist), not on files. And run GC in dry-run mode first: mine still runs with --defer-deletes, which only records what it would delete. Its log still says "deleted N files" in that mode, so check the bucket rather than the log line.

Pairing Iceberg with Nessie in a homelab is a nice way to get branch and merge semantics for data without paying for a managed catalog. One thing worth covering in a follow-up is snapshot expiration and orphan file cleanup, since on Ceph backed S3 those small files and old snapshots add up quickly if nobody schedules the maintenance jobs. A short section on how you back up the Nessie metadata store would also help anyone copying this setup.

Thanks, agreed on both. Two notes from running it.

Snapshot expiry works differently with Nessie. Iceberg tables behind Nessie keep a single snapshot in their own metadata, because the history lives in Nessie's commit log. So expire_snapshots has nothing to do, and the cleanup that matters is Nessie GC, which also deletes orphan files. Mine runs weekly with a 30-day cutoff, still in record-only mode. Compaction of small files isn't scheduled yet, and you're right that the table written every minute is the one that'll need it.

Nessie's metadata store is Postgres (the JDBC version store) on a 3-instance CloudNativePG cluster: continuous WAL archiving plus a nightly base backup to Ceph RGW, kept for 7 days. The honest gap: those backups sit on the same Ceph cluster as the data, so they cover losing a database, not losing the storage. An off-site copy is on my list. Good shout for a follow-up post.

Record-only mode first is a sensible way to build trust in the GC before it deletes anything. The off-site copy is the one I would prioritise, since a storage-level failure is exactly the case the current backups cannot cover. Looking forward to the follow-up.

Agreed, the off-site copy goes first. Thanks for pushing on it.

Search Hashnode

Self-Hosted Databricks Alternative: Homelab Data Lakehouse (AI/ML Data Factory)

Responses12