Tag

#inference

123 posts0 followers

Explore Hashnode

Alternatives

One Token Leaked 155 AI Responses: Inside Cloudflare's GLM-5.3 KV Cache Exposure

On September 8, 2026, a security researcher told Cloudflare that very long prompts sent to one of its hosted models sometimes came back with text from other customers' sessions. Cloudflare reproduced

damianobianda.hashnode.dev

Pratyush Ranjan5d ago · 8 min read

KV Cache Optimisation in AI Inference

When we chat with an AI tool like Claude or ChatGPT and await for a response, large mathematical operations happen behind the scenes. Most of these tools that we use today have models that are built o

1 ai-inference-pratyush.hashnode.dev

Adi Shik6d ago · 10 min read

What a 1997 chess engine taught me about LLM inference

TL;DR You don't need an LLM agent to optimize inference. Across six tuning tracks on an NVIDIA L4, Tensward's expert rules and heuristics matched frontier LLM agents running at maximum effort - Astra

openingthehood.hashnode.dev

Rounak ShowOct 23 · 14 min read

The Two Lives of an LLM Request: Prefill and Decode

The strange pause Okay, so lets say, you open a chat app and you paste a long document. Then, you ask, "Hey, can you summarize this?" You hit enter. Nothing happens. One second. Two seconds. Maybe thr

rounakshow.hashnode.dev

Sachin RajputOct 22 · 16 min read

One Request, End to End: Tokenize, Prefill, Decode, Stream

Corrections (updated Oct 29, 2026): the tiny engine in the first version of this post had two hidden slowdowns. One line (/ np.sqrt(self.hd)) quietly switched the math from 32-bit to slower 64-bit num

sachinrajput.hashnode.dev

RONI DASOct 17 · 8 min read

Why did one way of splitting Meta's 816B model run 59% faster on the same two Macs?

Why would two identical machines, running the same model, give you two different speeds? That is what Apple's machine learning team published on 15 October. They ran Muse Spark Open, the open-weights

systemdesignacademy.hashnode.dev

Akshay SuryawanshiOct 14 · 11 min read

A Database Engineer’s First Look at LLM Inference.

I’ve spent much of my career working with database infrastructure, especially MySQL. I’m now exploring LLM inference to find out which of those engineering instincts transfer, and I’m starting by runn

akshaysuryawanshi.hashnode.dev

Sanskriti HarmukhOct 14 · 25 min read

Configuring Smart Routing in NVIDIA Dynamo

NVIDIA Dynamo is an open-source, high-throughput, low-latency inference framework designed to deploy large-scale generative AI and reasoning models across multi-node, multi-GPU environments. Its KV Ro

vultr.hashnode.dev

kalyan dahakeSep 30 · 4 min read

I Built a Prefix-Cache-Aware LLM Gateway in Go — And Reduced TTFT by 90%

Most backend engineers treat LLM inference like standard microservices. You stand up an Nginx or AWS ALB round-robin load balancer, spin up multiple GPU workers running vLLM, Triton, or Ollama, and di

aekshant.hashnode.dev

Mir Nafis Sharear ShopnilAug 31 · 13 min read

How I Got Qwen3.8-27B from 18.66 to 192.40 tok/s on One L40S

We recently deployed Qwen3.8-27B internally at work. The setup was fairly straightforward: one NVIDIA L40S serving the model through vLLM, LiteLLM in front of it for authentication and per-user API ke

namikazi25.hashnode.dev

Search Hashnode