Discussion on "A hex literal made LLM code reviewers raise false alarms"

Sunny Chokshi · 2026-10-28T19:32:22.100Z

I wanted to know whether an LLM code reviewer follows what the code does or what the code looks like. Cryptographic API misuse is a clean place to ask, because the same insecure choice can be written

With a 3072-bit key written as 0xC00 getting flagged, I'd want to read the reviewer's stated reason next to the flag. If it says the key is too small, it misread the value. If it complains about a magic number, a style note got scored as a security finding. Did the false positives come with reasons you could sort that way?

Yes, I read the replies for these. For the two 3072-bit twins (0xC00 and 3 * 2 ** 10), 24 replies flagged them. All 24 decoded the value correctly as 3072. 21 then called it too small, weak or insufficient, two called it non-standard, and one cited the backend. So the models read the value and still judged it wrongly. Neither of your two cases fits exactly. It is a wrong stated reason, not a misread number. Sorting every false positive this way is the check I would add next.

Interesting that most of the false-alarm rise is on safe values written unusually, like 0xC00. For the open question at the end, a cheap ablation would be separating normalization from reasoning: pre-resolve constants with a deterministic step (an AST pass that folds literals and follows simple aliases), then hand the reviewer the folded value alongside the original code. If false positives still drop only for the folded variant and not for a prompt that merely says 'normalize values first', that would suggest the model can't do the arithmetic reliably rather than being ignorant of the spelling. Also, since two models flag nearly every snippet, it may be worth reporting results with a threshold or flag-rate-matched comparison so the indirection effect isn't confounded by base flag rates.

Useful design. Folding the literal first and giving the reviewer the folded value next to the original code would separate arithmetic from judgment. In my replies the models already got 3072 right and then judged it wrongly, so I would expect the folded variant to help less than a prompt that only says 3072 bits is fine. I have not tested that. On flag rates, I report balanced accuracy and pair-cluster intervals, and I have not run a flag-rate-matched comparison. I will note that as a limit.

That distinction matters: if the models decode 3072 correctly and then misjudge it, the failure is in the security-judgment step, not the arithmetic. It also suggests a third arm could help: give the reviewer the folded value plus an explicit reference (for example, 'RSA 3072 meets current minimum guidance') and check whether the 21 'too small' verdicts flip, which would test whether it's a missing-knowledge issue or a miscalibration one. Noting the flag-rate limit sounds like the right call.

Search Hashnode

A hex literal made LLM code reviewers raise false alarms

Responses5