[dead]
Two different tricks for fast LLM inference
71–80 of 82 posts
Re: Two different tricks for fast LLM inference
#72Earlier quoted context omitted.
So what you’re saying is that Cerebras chips offer 44GB of what is comparable to L1 caches, while NVidia is offering 80GB of what is comparable to “fast DRAM” ?
Sort of. But SRAM is not all made equal - L1 caches are small because they’re fast, and vice-versa L3 SRAM caches are slow because they’re big. To address a large amount of SRAM requires an approximately log(N) amount of logic just to do the addressing (gross approximation). That extra logic takes time for a lookup operation to travel through, hence large = slow. It’s also not one pool of SRAM. It’s thousands of smal…
Right. L3 caches, i.e. SRAMs of tens of MB or greater sizes have a latency that is only 2 to 3 times better than DRAM. SRAMs of only a few MB, like most L2 caches, may have a latency 10 times less than DRAM. L1 caches, of around 64 kB, may have a latency 3 to 5 times better than L2 caches.
The throughput of caches becomes much greater than of DRAM only when they are separated, i.e. each core has its private L1+L2 cache memory, so the transfers between cores and private caches can be done concurrently, without interference between them.
When an SRAM cache memory is shared, the throughput remains similar to that of external DRAM.
If the Cerebras memory is partitioned in many small blocks, then it would have low latency and high aggregate throughput for data that can be found in the local memory block, but high latency and low throughput for data that must be fetched from far away.
On the other hand, if there are fewer bigger memory blocks, the best case latency and throughput would be worse, but the worst case would not be so bad.
Re: Two different tricks for fast LLM inference
#73Article closes with: >The usefulness of AI agents is dominated by how few mistakes they make, not by their raw speed. Buying 6x the speed at the cost of 20% more mistakes is a bad bargain, because most of the user’s time is spent handling mistakes instead of waiting for the model6. That might be true today. I think the OpenAI-Cerebras partnership ultimately is going to lead to a paradigm shift because it will be poss…
[dead]
Generated comments and bots aren't allowed here, see: https://news.ycombinator.com/item?id=46888857
Re: Two different tricks for fast LLM inference
#74[deleted]
Re: Two different tricks for fast LLM inference
#75Earlier quoted context omitted.
Sharding the model is really slow. The point of building a wafer-scale chip is memory bandwidth for on-chip transfer is far more than you would get from even using chiplets with an interposer/high-bandwidth connection, let alone going off-chip. You're giving up your whole advantage, especially since Cerebras clearly isn't trying to maximize total throughput per watt - Groq, TPUs, and even the latest nVidia solutions…
There are ways to shard the model that require a lot of off-chip bandwidth, but there are also ways that don't. The only data that needs to be passed between layers is the residual stream, which requires much less bandwidth than the layer weights and KV cache, and you already need about that much bandwidth to get input tokens in and output tokens out. So putting different layers on different chips isn't that terrible…
Yeah but SRAM is stupidly fast. Even compared to DRAM. Going to a different chip, no matter if over a custom interconnect takes a "lifetime", specially since you don’t have L4 caching tricks, as you’re already at SRAM level.
Re: Two different tricks for fast LLM inference
#76Earlier quoted context omitted.
Yes this article is full of misunderstanding. The main explanation of bottleneck is wrong: it’s the model weights which dominate memory bandwidth (and hence why batching multiple requests in a single pass increases total throughput). If copying user tokens was the bottle neck, batching would not achieve any speed up. When an author is confused about something so elementary, I can’t trust anything else they write.
> The main explanation of bottleneck is wrong: it’s the model weights which dominate memory bandwidth (and hence why batching multiple requests in a single pass increases total throughput). If copy user tokens was the bottle neck, batching would not achieve any speed up. Inference is memory-bound only at low batch sizes. At high batch sizes it becomes compute-bound. There's a certain threshold where stuffing more req…
I would guess you haven't done this in practice. Yes, of course inference is memory bound at low batch sizes. This is why we run larger batch sizes!
Also there does not exist any batch size > 1 where per-request throughput is equal to bs=1. Doing any batching at all will slow all intra-batch requests down.
Re: Two different tricks for fast LLM inference
#77Earlier quoted context omitted.
Yes this article is full of misunderstanding. The main explanation of bottleneck is wrong: it’s the model weights which dominate memory bandwidth (and hence why batching multiple requests in a single pass increases total throughput). If copying user tokens was the bottle neck, batching would not achieve any speed up. When an author is confused about something so elementary, I can’t trust anything else they write.
They failed to grasp the very fundamental point of batching, which is sharing model weights between requests. For more context, this wasn't just one person's mistake, several AI twitter personalities proposed this 'Claude Opus fast = small batching' hypothesis. What I find funny is how confident these AI influencers were, while the people who actually work on LLM serving at frontier labs said nothing. The people who…
If you ask someone knowledgeable at r/LocalLLaMA about an inference configuration that can increase TG by *up to* 2.5x, in particularly for a sample prompt that reads "*Refactor* this module to use dependency injection", then the answer is of course speculative decoding.
You don't have to work for a frontier lab to know that. You just have to be GPU poor.
Re: Two different tricks for fast LLM inference
#78Earlier quoted context omitted.
[flagged]
Because inference is autoregressive (token n is an input for predicting token n+1), the forward pass for token n+1 cannot start until token n is complete. For a single stream, throughput is the inverse of latency (T = 1/L). Consequently, any increase in latency for the next token directly reduces the tokens/sec for the individual user.
Throughput is a metric for the total number of tokens/sec for all users in the system. Latency (ITL,TTFT) are individual user metrics.
Re: Two different tricks for fast LLM inference
#79In my view, if we aim to speed up progress toward the next generation, diffusion-model-based generation is a very promising direction.
That said, there are likely many hurdles to tackle, because the output is not a sequence; it is produced in parallel.
Re: Two different tricks for fast LLM inference
#80I like how the article is interspersed with corrections based on this discussion. Sometimes I wondered if there are more corrections than the original material. But in any case it is very convenient. No need to read the discussion here.