Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two different tricks for fast LLM inference

seangoedecke.com

Post reply on HN
Print view View original

41–50 of 82 posts

Re: Two different tricks for fast LLM inference

#41

Post by SkiFire13 »

BitsAndObjects wrote: post #27
stingraycharles wrote: post #26

Earlier quoted context omitted.

So what you’re saying is that Cerebras chips offer 44GB of what is comparable to L1 caches, while NVidia is offering 80GB of what is comparable to “fast DRAM” ?

Sort of. But SRAM is not all made equal - L1 caches are small because they’re fast, and vice-versa L3 SRAM caches are slow because they’re big. To address a large amount of SRAM requires an approximately log(N) amount of logic just to do the addressing (gross approximation). That extra logic takes time for a lookup operation to travel through, hence large = slow. It’s also not one pool of SRAM. It’s thousands of smal…

> L1 caches are small because they’re fast

I guess you meant to say they are fast because they are small?

Re: Two different tricks for fast LLM inference

#42

Post by mft_ »

BitsAndObjects wrote: post #21
mft_ wrote: post #14

> So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. That’s why they’re offering a brand new model, and why the Spark model has a bit of “small model smell” to it: it’s a smaller distil of the much larger GPT-5.3-C…

You’re mixing up HBM and SRAM - which is an understandable confusion. NVIDIA chips use HBM (High Bandwidth Memory) which is a form of DRAM - each bit is stored using a capacitor that has to be read and refreshed. Most chips have caches on them built out of SRAM - a feedback loop of transistors that store each bit. The big differences are in access time, power and density: SRAM is ~100 times faster than DRAM but DRAM…

Thanks, TIL.

Re: Two different tricks for fast LLM inference

#43

Post by zozbot234 »

yorwba wrote: post #12

> The idea is to have a chip with SRAM large enough to fit the entire model , so inference can happen entirely in-memory. [...] So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. You don't really need to fit the en…

Sharding the model is really slow. The point of building a wafer-scale chip is memory bandwidth for on-chip transfer is far more than you would get from even using chiplets with an interposer/high-bandwidth connection, let alone going off-chip. You're giving up your whole advantage, especially since Cerebras clearly isn't trying to maximize total throughput per watt - Groq, TPUs, and even the latest nVidia solutions are preferable there.

Re: Two different tricks for fast LLM inference

#44

Post by qudent »

BitsAndObjects wrote: post #24
qudent wrote: post #22

Earlier quoted context omitted.

It does affect the throughput for an individual user because you need all output tokens up to n to generate output token n+1

[flagged]

Because inference is autoregressive (token n is an input for predicting token n+1), the forward pass for token n+1 cannot start until token n is complete. For a single stream, throughput is the inverse of latency (T = 1/L). Consequently, any increase in latency for the next token directly reduces the tokens/sec for the individual user.

Re: Two different tricks for fast LLM inference

#45

Post by zozbot234 »

gchadwick wrote: post #13
qeternity wrote: post #9

Earlier quoted context omitted.

Yes this article is full of misunderstanding. The main explanation of bottleneck is wrong: it’s the model weights which dominate memory bandwidth (and hence why batching multiple requests in a single pass increases total throughput). If copying user tokens was the bottle neck, batching would not achieve any speed up. When an author is confused about something so elementary, I can’t trust anything else they write.

> If copying user tokens was the bottle neck, batching would not achieve any speed up. Reality is more complex. As context length grows your KV cache becomes large and will begin to dominate your total FLOPs (and hence bytes loaded). The issue with KV cache is you cannot batch it because only one user can use it, unlike static layer weights where you can reuse them across multiple users. Emerging sparse attention tec…

> The issue with KV cache is you cannot batch it because only one user can use it

This is not really correct given how input token caching works and the reality of subagent workloads. You could launch many parallel subagents sharing some portion of their input tokens and use batching for that task.

Re: Two different tricks for fast LLM inference

#49

Post by yorwba »

zozbot234 wrote: post #43
yorwba wrote: post #12

> The idea is to have a chip with SRAM large enough to fit the entire model , so inference can happen entirely in-memory. [...] So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. You don't really need to fit the en…

Sharding the model is really slow. The point of building a wafer-scale chip is memory bandwidth for on-chip transfer is far more than you would get from even using chiplets with an interposer/high-bandwidth connection, let alone going off-chip. You're giving up your whole advantage, especially since Cerebras clearly isn't trying to maximize total throughput per watt - Groq, TPUs, and even the latest nVidia solutions…

There are ways to shard the model that require a lot of off-chip bandwidth, but there are also ways that don't. The only data that needs to be passed between layers is the residual stream, which requires much less bandwidth than the layer weights and KV cache, and you already need about that much bandwidth to get input tokens in and output tokens out. So putting different layers on different chips isn't that terrible.

Importantly, Cerebras is offering many models that can't possibly fit on just a single chip, so they have to use some kind of sharding to get them to work at all. You could imagine an even bigger chip that can fit the entire model and run it even faster, but they have to work with what can be manufactured with current technology.

Re: Two different tricks for fast LLM inference

#50

Post by dan-robertson »

I think being faster probably is important but it brings a bunch of challenges:

- the split pricing model makes it hard to tune model architecture for faster inference as you need to support fast and cheap versions.

- the faster the model is, the more it becomes a problem that they don’t ’understand’ time – they sit idle waiting for big compilations or they issue tools sequentially when they ought to have issued them in parallel.

Post reply on HN