Discovering the inventory from the Proxmox API instead of hard-coding it is the best decision in here, and it quietly creates the one failure this design cannot see.
A job never names a host, it selects. So the scope of every job is a function of tags that live in a different repository and change for reasons unrelated to the job. Rename a tag in Terraform, drop one while refactoring a stack, or fat-finger it in a new guest's definition, and the job stops covering that guest. Nothing fails. nr exec fans out to eleven guests instead of twelve, every one of them succeeds, the job exits 0 and the Slack channel stays quiet. Silence is what you built the whole system to mean "everything is fine".
Founder & Automation Strategist @MageSheet | Google Apps Script & AI Specialist.
Ha, excellent catch! You're exactly right. Silence is the signal and a guest quietly falling out of a tag selector produces exactly that. It's softer than it sounds for the jobs I run today, since they select everything and opt guests out with exclude, so a new or renamed guest fails loudly. But the moment I write a tag-scoped job, your scenario is live. I'm adding an expect floor per job and a daily coverage job that lists any guest no job touches. I'll update the article on my blog accordingly and credit you for pointing this out. Thanks for this!
Select-all with exclude is a good default exactly because a new guest fails loudly, and I had missed that you were already doing it. The expect floor and the coverage job cover the two directions well: a job seeing fewer hosts, and a host no job sees. One small thing: if the coverage job stops running, it goes quiet the same way, so it is worth having it post a short all-clear line each day so its absence is noticed.
Hayrullah Kar
Discovering the inventory from the Proxmox API instead of hard-coding it is the best decision in here, and it quietly creates the one failure this design cannot see.
A job never names a host, it selects. So the scope of every job is a function of tags that live in a different repository and change for reasons unrelated to the job. Rename a tag in Terraform, drop one while refactoring a stack, or fat-finger it in a new guest's definition, and the job stops covering that guest. Nothing fails. nr exec fans out to eleven guests instead of twelve, every one of them succeeds, the job exits 0 and the Slack channel stays quiet. Silence is what you built the whole system to mean "everything is fine".