Some teams may assess Kafka's health incorrectly. They publish test messages to a topic. Or they spin up an AdminClient just to call DescribeClusterAsync(). Or they assume "no errors = healthy" and ho
Markdown supported · ⌘↵ to post
The "the health check itself made the cluster less reliable" pattern shows up far beyond Kafka, and your GC-pauses-to-restart-storm chain is the sharpest example of the monitor and the monitored sharing a fate. The failure mode I'd want a follow-up on: how do you notice when the piggybacked statistics themselves stop arriving? If the callback stalls silently, you end up with the calmest dashboard of all - the one that says nothing is wrong because nothing is being measured. Did you end up putting a watchdog on the absence of callbacks, or is staleness just accepted as a blind spot?
The restart storm section is the part every team rediscovers on its own. A health check that restarts the client turns one slow broker into a full fleet outage.
One more guard worth adding: wire this check to the readiness probe rather than liveness. A false unhealthy then pulls the pod out of service instead of restarting it, which avoids both the restart storm and the consumer rebalance the article describes.
The section on the stats callback firing while the broker is unreachable stood out to me — a health check that only looks at "did the callback fire recently" would happily report healthy during a disconnect. Parsing the broker connection state off-thread feels like the crucial half of this pattern that most write-ups skip.
Abhinav Sharma
Founder of Impact Gate. See who breaks before you change an API.
The restart-storm angle is the counterintuitive part, a health check that kills and restarts consumers can trigger the very rebalances that then look like downtime. Piggybacking on the statistics callback instead of publishing test messages also keeps the topics clean.