Really enjoyed this deep dive! What stood out to me is that you didn’t just focus on whether the model could make a decision—you actually looked at the less glamorous parts that matter in production: calibration, cold starts, prompt formatting, cost, and how the model behaves on data it hasn’t seen before.
The “declared mass” detail was especially interesting. It’s a great example of how a model can appear confident while actually responding to something different than what we intended.
I also appreciate the honesty around the limitations of the 4B model. Showing where the approach worked, where it failed to generalize, and why some of the improvements didn’t hold up on the held-out data makes the experiment much more useful than a simple “we built it and it works” post.
The privacy/control trade-off with running the model inside your own AWS environment is probably the biggest takeaway for me. Great technical write-up!
B
BotSailor
Building the infrastructure for omnichannel automation. Insights on APIs, webhooks, and scaling AI-driven customer journeys.
Thanks, glad it came across that way! The held-out rows were the most useful part to write, even if they weren't the most flattering 😅 And agreed, keeping the decision inside your own account is the real selling point here.
Prompt length is a tax when the model already knows the repo layout. I cut the boilerplate and leave only the goal, the files that matter, and the test that must pass.
Fully agree, that's the design principle here. The one larger prompt is the last step, after every earlier check has passed, and it only gets the trace and the files the frames name: the minimum needed to write the issue.
Really enjoyed this deep dive! What stood out to me is that you didn’t just focus on whether the model could make a decision—you actually looked at the less glamorous parts that matter in production: calibration, cold starts, prompt formatting, cost, and how the model behaves on data it hasn’t seen before.
The “declared mass” detail was especially interesting. It’s a great example of how a model can appear confident while actually responding to something different than what we intended.
I also appreciate the honesty around the limitations of the 4B model. Showing where the approach worked, where it failed to generalize, and why some of the improvements didn’t hold up on the held-out data makes the experiment much more useful than a simple “we built it and it works” post.
The privacy/control trade-off with running the model inside your own AWS environment is probably the biggest takeaway for me. Great technical write-up!