Building an LLM Inference Engine with AI: The Code Was the Easy Part

Reading this the day after Cloudflare published their GLM-5.3 postmortem makes the title land even harder. The part that was "easy" — generating tokens — worked fine from day one. What leaked 155 responses across tenants was everything around it: a KV pool sized from leftover GPU memory (681K tokens) while the advertised context was 1M, and a rejection path that swapped the over-length prompt for a one-token placeholder instead of actually stopping the request, so the decode node generated from stale cache pages that still held other users' sessions. The two lessons they list map directly onto your point about understanding the system: they validated HTTP status codes at long contexts but never the output's coherence, and nothing forced the pool size and the context length to agree — now they check it from the running engine before every deploy. The code really was the easy part.