Discussion on "I Told Codex to Test Everything. It Believed Me."

Kerry Kier · 2026-10-28T21:05:45.220Z

When I finally had Claude go through three weeks of Codex's hardening work, its summary was blunt enough that I'm only paraphrasing a little: I was hardening a children's book website against NSA-leve

161 subagents on September 26 alone, two days after you switched to GPT-6 Sol, and by then there was no way to tell whether the model or the P50 and P51 rounds drove it. I'd put the subagent count into those three plain daily sentences too. A burst like that should get flagged the same day.

I am not a develper and it shows, I am a sys engineer and build infrastructure so this was a large part of not knowing. I also think a small part was the fact i was using Daybreak with some Easing of cyber controls so it worked a bit more than a standard model. As for the model change i learned rthe hard way never to do this, and you are right i dont know if it was the OP50/51 rounds or the model that caused the issue, but apples endpoint security part was the ultiimate block. I could have taken a different direction and not used the enpoint piece of it but i would have lost the past 2 weeks of work and would have had to start over, in in a session that ran for a total of 14+ Days of actual agent work, would have been a nightmare.

That 14-day session is the strongest argument for checkpoints. Since the Apple endpoint-security piece was the real blocker, a short written decision log at each phase boundary (what changed, why, what the exit criterion was) would let you retrace whether the model switch or the extra rounds drove the growth, and make it cheaper to branch off without losing the two weeks of work. Pinning the model for the length of a campaign and changing it only at a boundary is also a simple way to keep the comparison clean next time.

The capybara site versus 443 subagents is what got me. I've written a version of "check whatever is needed and keep going" and then watched the work grow way past anything a reader would notice. A hard stop before I walk away would have saved me the same kind of week.

You nailed this on the head, I shoud have gave it more controls, becuase ultimately when claude looked it over it came back and said basically, you are testing and hardening based on foregn giovernments trying to compromise the site and in reality, those people are not attacking a independant book webiste. Now what I have done is i have an agent that monitors CVE's and KEV's and as soon as one is relaeased (there is an hourly check for this) it tells the maintence and security subagents to investigate and supply a report via email with the reocommendation and the ability for me to either approve or instracut via email. So i figured this along with monthly maintenance on the server shoudl have most of the basis covered. I also run this site and one of my others on my own Infrastruture at home and putit behind cloudfalre so i did not have to open up ports on the firewall.

The "whatever is needed" sentence is the real finding here, and I'd go a step further: the day-one scope prompt could also include a stop condition that's checked mechanically, not just stated. For example, a fixed budget of subagents or wall-clock hours per phase, after which the agent must stop and report rather than propose another revision. A related suggestion is to make the agent rank findings by cost-to-fix versus severity before designing anything, so the three Medium host findings and the absolute_redirect off; fix land first. One question: since each campaign was bound to an exact artifact, did you consider a cheaper tier of evidence (a smoke subset of the 247 cases) for revisions that only touch files unrelated to the failure paths?

After running this i learmned a lot more on how to approach this and now with artifacts avaiible as well as having it attached to my internal git with all the documents and instructions made the next hardening much easier and claude was able to do it with CVP entitlement with Mythos 5.1 a lot faster and mythos before i started even stated it did a full scan oin the server to see if that much was even needed. I have a global prompt that does not agree with me and doesnt try to appease but will go thorugh and think of ouitcomes before it answers as well as some other controls, becuase i got tired of it telling me i was right all the time when i was clearly wrong.

Search Hashnode

I Told Codex to Test Everything. It Believed Me.

Responses7