Introduction When people talk about AI hardware, the conversation usually centers on compute. How many teraflops can a chip push? How many operations per second? Faster chips sound like faster AI. But
Markdown supported · ⌘↵ to post
Nice framing of the two bandwidths. A useful rule of thumb for readers: for single-user decode, tokens/sec is roughly memory bandwidth divided by the bytes of weights read per token. So a 70B model at 4-bit (~40GB) on ~270 GB/s LPDDR5X tops out around 6-7 tok/s in theory, regardless of how much compute or capacity you have. That's why unified memory is a big win for fitting models and fine-tuning, while MoE models (only active experts read per token) and speculative decoding are what actually make big models feel fast on these boxes. Prefill, by contrast, is compute-bound, so long RAG contexts behave very differently from generation.
Puneet Khandelwal
Local inference runs straight into a silent killer right now: memory bandwidth. Architectures that attack this exact bottleneck are the ones that actually matter.