The 74-Point Accuracy Gap: Why AI Analytics Fails Without Human Curation (And How to Fix It)

Great question, and you're pointing at the exact layer that's easiest to forget.

From what Anthropic shared, the semantic layer does live in the same repo as everything else — they say "nearly all data code (modeling, semantic layer, reference docs, canonical dashboard definitions)" is colocated, with CI checks that protect cross-layer integrity. So if a modeling change would break a downstream metric definition, CI should flag it.

But I think your instinct is right that there's a gap between "CI catches a breaking schema change" and "CI catches a metric definition that's now subtly wrong because the business meaning shifted."

A renamed column breaks the build. A table that silently starts excluding a new customer segment doesn't. That second kind of drift is exactly what you're describing — the artifact everyone assumes is stable because a human signed off on it once.

Anthropics's mitigation seems to be the offline eval set — they run it in CI so a PR touching a dependency re-runs the affected evals, and they track the share of agent queries that resolve through the semantic layer as a production signal. But they also explicitly say the silent failure mode (answer is wrong but looks plausible) is one they don't have a robust solution for yet.

I'd be curious how you're approaching this in your evidence-and-provenance work. Are you testing the metric output itself (does revenue still match the blessed dashboard), or testing the definition's validity against the underlying schema?

Both, but they catch different failure modes so neither one alone covers it. We test the metric output against a small set of reference points we've manually reconciled once and pinned as a regression baseline, since that catches drift even when nobody can say why the definition changed. But that only tells you the number moved, not why, so we pair it with a check on the definition's dependencies themselves: does every table and join the metric relies on still have the same grain and null behavior it had when a human last signed off, and if a dependency's shape changes the metric gets flagged for re-review instead of trusting it silently held. The failure mode we've actually hit is the same one Anthropic names, an answer that's wrong but plausible, and the dependency check is what catches it before the reference-point comparison would show visible drift, because a segment that's quietly excluded doesn't always move an aggregate enough to trip a naive value comparison.