A Go test that passed 40,000 times, then lost a write under load

Say a client writes a value and gets a success back. Later, the value is gone. It happens once in production, around the time a node restarts, and every test is green. A week later a CI job fails on a