Running a Coding Agent on a Local GLM 4.7 with vLLM: The Setup Nobody Documents
Giving a coding agent its own brain for zero API cost is genuinely possible now. GLM 4.7 Flash serves on two consumer RTX 3090s, and a real agent framework can point at it instead of a cloud endpoint.
Responses
Markdown supported · ⌘↵ to post
Timely companion read: Cloudflare's postmortem from yesterday about their hosted GLM-5.3. Their bug wasn't the model, it was the serving config: the prefill KV pool sized itself from leftover GPU memory to 681K tokens while the advertised context was 1M, and nothing forced the two to agree. Over-length prompts got "rejected" but kept flowing through a disaggregated pipeline as one-token placeholders, so responses attended over stale cache pages from other users' sessions. Two transferable checks for anyone self-hosting GLM on consumer cards: verify the running engine's actual KV pool size against the context length you serve (from the engine's own metrics, not from a green HTTP 200), and treat any mismatch as a live bug, not a performance footnote. Single-tenant local setups dodge the cross-tenant exposure entirely — one more point for the setup you're describing.