The schema-as-contract point connects directly to what I hit on the application side: validating data at the edges only works if the contract is treated as public API, not as an implementation detail someone can rename on a Tuesday. The idempotency section is the one I'd underline twice. The consumers that break are never the ones you expect - it's the quiet retry path somebody adds six months later that violates idempotency without anyone noticing. Did you end up enforcing it with a dedupe store keyed on event ID, or did you push it down into the consumer logic itself?
The dead-letter-as-a-mirror point is spot on. Something that helped us: alert on the age of the oldest message in the DLQ, not just the count, because a slow trickle looks harmless by count but turns into data loss once it passes the retention window. And for the Redis dedup layer, make sure the key TTL is longer than your longest possible redelivery gap (including replaying from the DLQ), otherwise a replay days later skips dedup and you get the duplicate orders again.
The distinction between acknowledgement and correct output is useful. I would make the processed-per-minute metric count each event_id once and chart delivery attempts separately. During a restart storm, higher throughput can otherwise mean duplicate work. Do your data-quality checks also reconcile expected distinct events with completed downstream actions?
When a consumer processes an event successfully but fails before acknowledging it, the message may be delivered again. How do you design idempotency and reconciliation together to prevent duplicate downstream updates while ensuring that no events are lost during retries or dead-letter queue reprocessing?
How did you handle the gap between recording an event_id in Redis and completing the downstream action? If a worker dies between those steps, marking it first can skip unfinished work; marking it afterward can repeat an email. Did you use a processing/completed state or downstream idempotency keys, and did the dedup retention cover dead-letter replays? (AI-assisted comment.)
The Redis deduplication layer keyed on event_id is a good example of a small fix with a big payoff, especially since the duplicated orders and double emails came from an ordinary deployment restart. Alerting on dead-letter depth above zero rather than a threshold is also a sharp rule, since three days of silent drops is exactly what a threshold hides. And watching oldest_unacked_message_age instead of raw backlog size is a useful way to tell a draining backlog from a consumer that is truly falling behind.
Julian Neagu
500+ AI tools shipped solo. Founder of VisionVix.
The dead-letter queue point hits home. I've seen how easy it is to assume things are fine just because the main service looks healthy. Checking what actually made it through matters just as much.