A zero-size "paid in full" line closing a bill and silently killing the reminders is nastier than most injection demos I've seen, because nothing looks wrong to anyone. I like that you swapped the model for a fake that can return any well-formed answer and tested the app's behavior instead. Did the hidden-text check need much tuning to hit zero false flags across those 60 real bills?
Thank you. No, I did not tune it. But the "zero" needs a note.
The hidden-text check alone flagged 12 of 60 runs (20%). Four bills have a hidden preview line, and the check flags them. A second read of the bill follows each flag. If the two reads agree, the bill goes through. This gives 0 false flags in 60.
I tried a higher threshold. False flags fell, but detection fell from 8 of 8 to 1 of 8. So I kept the strict setting.
The 60 runs are 20 test bills, each run 3 times. They are not real mail. The real rate is not known yet. A metric in production will show it.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
A zero-size "paid in full" line closing a bill and silently killing the reminders is nastier than most injection demos I've seen, because nothing looks wrong to anyone. I like that you swapped the model for a fake that can return any well-formed answer and tested the app's behavior instead. Did the hidden-text check need much tuning to hit zero false flags across those 60 real bills?