Brownfield Kubeflow on RKE2: Swapping Dex & SeaweedFS for Keycloak & Ceph RGW

Thanks, and yes, that's exactly the trap. On the Pipelines side it bit me twice. The kfp-launcher config needs an explicit providers block for S3, or artifact uploads quietly go nowhere, and region is mandatory even though RGW doesn't care about regions. Then every component has to use the same endpoint (the public RGW hostname on 443, not the in-cluster service on 80), or some steps write where others can't read.

The smoke test you describe runs every day here, not just after upgrades. The daily pipeline trains the model, writes it to RGW, rolls it out to KServe as a canary, and sends a burst of real requests at it before promoting. If KServe can't pull the artifact, that step fails and the old model keeps serving. The same pipeline also turned up a KServe-specific one: any extra .json file next to model.bst in the model folder crashes the XGBoost server at load, so the model folder holds only the model.