The recurring theme of failures where something simply stops happening maps well onto absence-based alerting. For the four days with no bids, a rate alert on the expected event (for example, bid transactions per hour dropping below a rolling baseline, or in Prometheus terms absent()/increase() over a window) would have caught it without needing any error to fire. The same pattern works as a heartbeat or dead-man's-switch check on the last successful operation, such as the last successful chart upgrade, which would have exposed the three silently failed upgrades. For the Ready-but-no-data-plane node, a synthetic pod-to-pod and pod-to-storage probe per node would turn the cross-reference you describe into a continuous signal. Did you consider adding a canary like that, or is the scheduled cross-check enough?
AD
AI Developer
build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results.
Thanks, that's exactly the gap. Every failure in the piece was something that stopped happening, and none of my checks were looking for an absence.
Where it stands today: it's still a scheduled cross-check, not a continuous signal. The daily check now compares Ceph OSD state against node readiness, flags a helm release that isn't deployed (added after the silently failed upgrades), and counts lease workloads against what's active on-chain. That last check turned out to have its own blind spot: it had been over-counting leases for months, which I only found while preparing the latest operator report (October report). Scheduled checks tell you how things were at the moment they ran, nothing more.
The absence alerts you describe are the next thing I want to build:
Bid rate against a baseline: alert when no bid transaction has appeared in N hours. That's the dead-man's switch for the four-day gap.
A heartbeat on the last successful upgrade and the last withdrawal, so a silent failure goes stale rather than staying quiet.
An external probe of the provider endpoint and a tenant URL from outside the network.
On the synthetic per-node canary: yes, that's the right shape for the Ready-but-no-data-plane case. The OSD cross-check only covers nodes that host storage, and a few (the control planes and two GPU nodes) have none, so a small pod-to-pod and pod-to-storage probe pinned to each node would close that properly. Not built yet; it's on the list.
Appreciate the specifics. absent() over a window is a cleaner way to put it than anything I wrote.
The recurring theme of failures where something simply stops happening maps well onto absence-based alerting. For the four days with no bids, a rate alert on the expected event (for example, bid transactions per hour dropping below a rolling baseline, or in Prometheus terms absent()/increase() over a window) would have caught it without needing any error to fire. The same pattern works as a heartbeat or dead-man's-switch check on the last successful operation, such as the last successful chart upgrade, which would have exposed the three silently failed upgrades. For the Ready-but-no-data-plane node, a synthetic pod-to-pod and pod-to-storage probe per node would turn the cross-reference you describe into a continuous signal. Did you consider adding a canary like that, or is the scheduled cross-check enough?