Discussion on "My website had the default wrong for 86 days"

Brock Claussen · 2026-10-19T13:00:00.000Z

Foxstash 0.9 shipped on Saturday. The release notes came to me from the Claude Code session that maintains the library, written to be copied onto my company's website. Before any of it went on the sit

Checking each claim against the code is underrated for release notes - docs drift from behavior quietly, and a wrong default label survives for months because nobody runs the default path again.

The distinction between a default and a preset is easy to overlook, but it can change the behavior of an application without anyone realizing why. I'd take this a step further and make the capabilities register part of the release pipeline.

For example, if Foxstash changes its default storage mode from F32 to SQ8, a documentation test could compare the published default against the actual Default implementation or configuration resolution path at the release tag. Benchmark claims could be tracked separately, with the exact mode pair, dimensionality, dataset, and measured memory component attached to each result. That would make it harder for a valid number to end up supporting the wrong claim.

Have you considered enforcing those checks in CI, so a release with stale defaults or untraceable benchmark claims can't be published in the first place?

Not in CI yet. Today the check is done by hand against the code at the release tag, before anything goes on the site. You're right that the default is the easy one to automate, since a test can read the actual Default and compare. Benchmark claims are harder. The failure here wasn't a stale number but a real number attached to the wrong pair, so what I'd want is the measured pair, dataset and dimensions stored next to each number, so a sentence can only cite what was measured.

The other thing I'd watch is whether the benchmark result captures the conditions that made the number valid in the first place. With ANN search, a result can be perfectly reproducible and still support the wrong conclusion if the comparison used a fixed ef_search instead of matched recall, or if one run retained full-precision vectors for reranking while the other didn't. I'd tie each result to the commit SHA, compiler flags, CPU/thread configuration, distance metric, and reranking settings, then have the report generator reject comparisons that aren't equivalent on the relevant dimensions. For something like 960-d GIST, I'd also keep the recall/QPS trade-off visible rather than reducing the result to a single speedup figure. Otherwise, the metadata can be correct while the headline still tells the wrong story.

Search Hashnode

My website had the default wrong for 86 days

Responses4