Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Two different tricks for fast LLM inference

seangoedecke.com

Post reply on HN
Print view View original

21–30 of 82 posts

Re: Two different tricks for fast LLM inference

#21

Post by BitsAndObjects »

mft_ wrote: post #14

> So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. That’s why they’re offering a brand new model, and why the Spark model has a bit of “small model smell” to it: it’s a smaller distil of the much larger GPT-5.3-C…

You’re mixing up HBM and SRAM - which is an understandable confusion.

NVIDIA chips use HBM (High Bandwidth Memory) which is a form of DRAM - each bit is stored using a capacitor that has to be read and refreshed.

Most chips have caches on them built out of SRAM - a feedback loop of transistors that store each bit.

The big differences are in access time, power and density: SRAM is ~100 times faster than DRAM but DRAM uses much less power per gigabyte, and DRAM chips are much smaller per gigabyte of stored data.

Most processors have a few MB of SRAM as caches. Cerebras is kind of insane in that they’ve built one massive wafer-scale chip with a comparative ocean of SRAM (44GB).

In theory that gives them a big performance advantage over HBM-based chips.

As with any chip design though, it really isn’t that simple.

Re: Two different tricks for fast LLM inference

#22

Post by qudent »

amelius wrote: post #20
yorwba wrote: post #12

> The idea is to have a chip with SRAM large enough to fit the entire model , so inference can happen entirely in-memory. [...] So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. You don't really need to fit the en…

> Of course when you have a long pipeline of chips that each token needs to pass through, that decreases the end-to-end tokens per second correspondingly. No, it only increases the latency, and does not affect the throughput.

It does affect the throughput for an individual user because you need all output tokens up to n to generate output token n+1

Re: Two different tricks for fast LLM inference

#23

Post by BitsAndObjects »

amelius wrote: post #20
yorwba wrote: post #12

> The idea is to have a chip with SRAM large enough to fit the entire model , so inference can happen entirely in-memory. [...] So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. You don't really need to fit the en…

> Of course when you have a long pipeline of chips that each token needs to pass through, that decreases the end-to-end tokens per second correspondingly. No, it only increases the latency, and does not affect the throughput.

It affects both. These systems are vastly more complex than the naive mental models being discussed in these comments.

For one thing, going chip-to-chip is not a faultless process and does not operate at the same speed as on-chip communication. So, yes, throughput can be reduced by splitting a computation across two chips of otherwise equal speed.

Re: Two different tricks for fast LLM inference

#24

Post by BitsAndObjects »

qudent wrote: post #22
amelius wrote: post #20

Earlier quoted context omitted.

> Of course when you have a long pipeline of chips that each token needs to pass through, that decreases the end-to-end tokens per second correspondingly. No, it only increases the latency, and does not affect the throughput.

It does affect the throughput for an individual user because you need all output tokens up to n to generate output token n+1

[flagged]

Re: Two different tricks for fast LLM inference

#25

Post by BitsAndObjects »

johndough wrote: post #16
yorwba wrote: post #12

> The idea is to have a chip with SRAM large enough to fit the entire model , so inference can happen entirely in-memory. [...] So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. You don't really need to fit the en…

> So the size of GPT-5.3-Codex-Spark isn't limited by the memory of a single Cerebras chip, but the number of such chips that you can chain together and still hit the 1000 tokens per second target. Chaining chips does not decrease token throughput. In theory, you could run models of any size on Cerebras chips. See for example Groq's (not to be confused with Grok) chips, which only have 230 MB SRAM, yet manage to run…

Only if chip-to-chip communication is as fast as on-chip communication. Which it isn’t.

Re: Two different tricks for fast LLM inference

#26

Post by stingraycharles »

BitsAndObjects wrote: post #21
mft_ wrote: post #14

> So how much internal memory does the latest Cerebras chip have? 44GB. This puts OpenAI in kind of an awkward position. 44GB is enough to fit a small model (~20B params at fp16, ~40B params at int8 quantization), but clearly not enough to fit GPT-5.3-Codex. That’s why they’re offering a brand new model, and why the Spark model has a bit of “small model smell” to it: it’s a smaller distil of the much larger GPT-5.3-C…

You’re mixing up HBM and SRAM - which is an understandable confusion. NVIDIA chips use HBM (High Bandwidth Memory) which is a form of DRAM - each bit is stored using a capacitor that has to be read and refreshed. Most chips have caches on them built out of SRAM - a feedback loop of transistors that store each bit. The big differences are in access time, power and density: SRAM is ~100 times faster than DRAM but DRAM…

So what you’re saying is that Cerebras chips offer 44GB of what is comparable to L1 caches, while NVidia is offering 80GB of what is comparable to “fast DRAM” ?

Re: Two different tricks for fast LLM inference

#27

Post by BitsAndObjects »

stingraycharles wrote: post #26
BitsAndObjects wrote: post #21

Earlier quoted context omitted.

You’re mixing up HBM and SRAM - which is an understandable confusion. NVIDIA chips use HBM (High Bandwidth Memory) which is a form of DRAM - each bit is stored using a capacitor that has to be read and refreshed. Most chips have caches on them built out of SRAM - a feedback loop of transistors that store each bit. The big differences are in access time, power and density: SRAM is ~100 times faster than DRAM but DRAM…

So what you’re saying is that Cerebras chips offer 44GB of what is comparable to L1 caches, while NVidia is offering 80GB of what is comparable to “fast DRAM” ?

Sort of. But SRAM is not all made equal - L1 caches are small because they’re fast, and vice-versa L3 SRAM caches are slow because they’re big.

To address a large amount of SRAM requires an approximately log(N) amount of logic just to do the addressing (gross approximation). That extra logic takes time for a lookup operation to travel through, hence large = slow.

It’s also not one pool of SRAM. It’s thousands of small SRAM groups spread across the chip, with communication pathways in between.

So to have 44GB of SRAM is a very different architecture to 80GB of (unified) HBM (although even then that’s not true as most chips use multiple external memory interfaces).

HBM is high bandwidth. Whether that’s “fast” or not depends on the trade off between bandwidth and latency.

So, what I’m saying is this is way more complicated than it seems. But overall, yeah, Cerebras’ technical strategy is “big SRAM means more fast”, and they’ve not yet proven whether that’s technically true nor whether it makes economic sense.

Re: Two different tricks for fast LLM inference

#28

Post by villgax »

Lol, without any evidence this is just vaporblog, it could just be reudced precision for whatever model either one of them runs & not necessarily a distillation or smaller model to boot or heck even a combo since at this point in time most frontier models are MoEs & getting absurd speeds for 1-20B experts is trivial regardless of batch sizes
Post reply on HN