Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two different tricks for fast LLM inference

seangoedecke.com

Post reply on HN
Print view View original

31–40 of 82 posts

Re: Two different tricks for fast LLM inference

#31

Post by xcodevn »

qeternity wrote: post #9
dist-epoch wrote: post #6

The batch size explanation is wrong. Given how much Claude Code is used, finding fellow "bus passengers" is not an issue, you don't need to wait. The real reason which batching increases latency is multi-factored and more complex to explain.

Yes this article is full of misunderstanding. The main explanation of bottleneck is wrong: it’s the model weights which dominate memory bandwidth (and hence why batching multiple requests in a single pass increases total throughput). If copying user tokens was the bottle neck, batching would not achieve any speed up. When an author is confused about something so elementary, I can’t trust anything else they write.

They failed to grasp the very fundamental point of batching, which is sharing model weights between requests. For more context, this wasn't just one person's mistake, several AI twitter personalities proposed this 'Claude Opus fast = small batching' hypothesis. What I find funny is how confident these AI influencers were, while the people who actually work on LLM serving at frontier labs said nothing. The people who genuinely understand this and work at frontier labs stay quiet. The rest is simply noise.

Re: Two different tricks for fast LLM inference

#32

Post by littlestymaar »

BitsAndObjects wrote: post #24
qudent wrote: post #22

Earlier quoted context omitted.

It does affect the throughput for an individual user because you need all output tokens up to n to generate output token n+1

[flagged]

How do you think that works?!

With the exception of diffusion language models that don't work this way, but are very niche, language models are autoregressive, which means you indeed need to process token in order.

And that's why model speed is such a big deal, you can't just throw more hardware at the problem because the problem is latency, not compute.

Re: Two different tricks for fast LLM inference

#33

Post by littlestymaar »

BitsAndObjects wrote: post #25
johndough wrote: post #16

Earlier quoted context omitted.

> So the size of GPT-5.3-Codex-Spark isn't limited by the memory of a single Cerebras chip, but the number of such chips that you can chain together and still hit the 1000 tokens per second target. Chaining chips does not decrease token throughput. In theory, you could run models of any size on Cerebras chips. See for example Groq's (not to be confused with Grok) chips, which only have 230 MB SRAM, yet manage to run…

Only if chip-to-chip communication is as fast as on-chip communication. Which it isn’t.

It doesn't need to, during inference there's little data exchange between one chip and another (just a single embedding vector per token).

It's completely different during training because of the backward pass and weight update, which put a lot of strain on the inter-chip communication, but during inference even x4 PCIe4.0 is enough to connect GPUs together and not lose speed.

Re: Two different tricks for fast LLM inference

#34

Post by woadwarrior01 »

aurareturn wrote: post #19
mft_ wrote: post #14

> So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. That’s why they’re offering a brand new model, and why the Spark model has a bit of “small model smell” to it: it’s a smaller distil of the much larger GPT-5.3-C…

It does make sense. Nvidia chips do not promise 1,000+ tokens/s. The 80GB is external HBM, unlike Cerebras’ 44GB internal SRAM. The whole reason Cerebras can inference a model thousands of tokens per second is because it hosts the entire model in SRAM. There are two possible scenarios for Codex Spark: 1. OpenAI designed a model to fit exactly 44GB. 2. OpenAI designed a model that require Cerebras to chain multiple wa…

Let's not forget the KV-cache which needs a lot of RAM too (although not as much as the model weights), and scales up linearly with sequence length.

Re: Two different tricks for fast LLM inference

#35

Post by kittbuilds »

BitsAndObjects wrote: post #27
stingraycharles wrote: post #26

Earlier quoted context omitted.

So what you’re saying is that Cerebras chips offer 44GB of what is comparable to L1 caches, while NVidia is offering 80GB of what is comparable to “fast DRAM” ?

Sort of. But SRAM is not all made equal - L1 caches are small because they’re fast, and vice-versa L3 SRAM caches are slow because they’re big. To address a large amount of SRAM requires an approximately log(N) amount of logic just to do the addressing (gross approximation). That extra logic takes time for a lookup operation to travel through, hence large = slow. It’s also not one pool of SRAM. It’s thousands of smal…

[dead]

Re: Two different tricks for fast LLM inference

#36

Post by bob1029 »

BitsAndObjects wrote: post #11

This author thinks Cerebras chips were deployed at scale to serve users worldwide in just one month since the partnership announcement? Seems like nonsense to me.

Did the author claim this?

OpenAI and Cerebras have been working together at some level for nearly a decade.

Re: Two different tricks for fast LLM inference

#37

Post by johndough »

BitsAndObjects wrote: post #25
johndough wrote: post #16

Earlier quoted context omitted.

> So the size of GPT-5.3-Codex-Spark isn't limited by the memory of a single Cerebras chip, but the number of such chips that you can chain together and still hit the 1000 tokens per second target. Chaining chips does not decrease token throughput. In theory, you could run models of any size on Cerebras chips. See for example Groq's (not to be confused with Grok) chips, which only have 230 MB SRAM, yet manage to run…

Only if chip-to-chip communication is as fast as on-chip communication. Which it isn’t.

Only if chip-to-chip communication was a bottleneck. Which it isn't.

If a layer completely fits in SRAM (as is probably the case for Cerebras), you only have to communicate the hidden states between chips for each token. The hidden states are very small (7168 floats for DeepSeek-V3.2 https://huggingface.co/deepseek-ai/DeepSeek-V3.2/blob/main/c... ), which won't be a bottleneck.

Things get more complicated if a layer does not fit in SRAM, but it still works out fine in the end.

Re: Two different tricks for fast LLM inference

#38

Post by x_may »

anonymous908213 wrote: post #8
Der_Einzige wrote: post #5

Another possible explanation, especially if quality degrades at all (I.e on openAI) is aggressive quantization. Another possible explanation is speculative decoding, where you trade unused GPU memory for speed (via a drafting model). But my money is on the exact two mechanisms the OP proposes.

> especially if quality degrades at all It is worth noting that consumers are completely and totally incapable of detecting quality degradation with any accuracy. Which is a given since the models are already effectively random, but there is a strong bent to hallucinate degradations. Having done frontend work for an AI startup, complaints of degrading the model were by far the most common, despite the fact that not o…

Wait sorry how did you use and expose seeds? That’s the most interesting part of your post

Re: Two different tricks for fast LLM inference

#39

Post by catoc »

BitsAndObjects wrote: post #24
qudent wrote: post #22

Earlier quoted context omitted.

It does affect the throughput for an individual user because you need all output tokens up to n to generate output token n+1

[flagged]

Your comment may be helpful - but would be much more helpful if you shared how it does work.

Edit: I see you’ re doing this further down; #thumbs up

Post reply on HN