Good catch on the idle-machine rerun, that is the benchmarking habit most comparisons skip. One caveat on the Ollama default: the 4,096 context it picked is much smaller than llama.cpp's per-slot context, so long prompts get silently truncated unless you raise OLLAMA_CONTEXT_LENGTH or set num_ctx. Worth noting if someone compares answer quality and not just tokens per second. Also, a first request after model load includes load time, so warming the model with a throwaway prompt keeps the single-request number clean.
Good catches, both worth flagging. You're right that Ollama's context default scales with VRAM (4,096 under ~24GiB, which is exactly where most single-developer/laptop setups land — the use case this article points people toward for Ollama), and it truncates silently rather than erroring, so a benchmark can look clean on short prompts and quietly degrade on longer ones. OLLAMA_CONTEXT_LENGTH is the right lever for a server-wide default; num_ctx overrides it per-request or in the Modelfile. The warm-up point is well taken too — cold model-load time folded into a single request is a classic way to make steady-state throughput look worse than it is. Might work both into a future revision of this piece.
Yunus Emre Vurgun
Writing about AI agents, hardware, and practical software.
Good catch on the idle-machine rerun, that is the benchmarking habit most comparisons skip. One caveat on the Ollama default: the 4,096 context it picked is much smaller than llama.cpp's per-slot context, so long prompts get silently truncated unless you raise OLLAMA_CONTEXT_LENGTH or set num_ctx. Worth noting if someone compares answer quality and not just tokens per second. Also, a first request after model load includes load time, so warming the model with a throwaway prompt keeps the single-request number clean.