How a do-not-disrupt annotation broke Karpenter consolidationKarpenter launched 2,800 nodes over one weekend and consolidated only 180 of them, and our EC2 on-demand bill jumped 2.2x. The cause was one line added to a shared internal Helm chart on Friday aftern1Aug 18 · 13 min readblog.infraforge.agency
Worker drops jobs intermittently: startup race, env drift, schema skewIf your worker is dropping jobs intermittently, sometimes crashing at boot with 'Error 111 connecting to redis:6379. Connection refused' and sometimes producing an empty result.json with no error at aAug 10 · 11 min readblog.infraforge.agency
How a missing S3 gateway endpoint route quintupled our NAT billThe finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump 1Jul 16 · 8 min readblog.infraforge.agency
How one stuck PDB doubled our EKS autoscaler bill in six daysThe AWS bill for EKS Compute went from $4,200 to $9,800 in six days. Same cluster, no launch, no traffic event, no team asking for headroom. kubectl get nodes returned 60 m5.4xlarge instances where thJul 16 · 12 min readblog.infraforge.agency
Recovering a status page from a half-finished schema migrationThe log line was 'database schema version 23 is dirty, refusing to start' and the pod exited immediately after printing it. The team had already tried a Helm rollback to the previous chart version. ThJul 13 · 11 min readblog.infraforge.agency