Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

cerebras.ai

Post reply on HN
Print view View original

71–80 of 100 posts

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#71

Post by moralestapia »

acchow wrote: post #66
moralestapia wrote: post #60

Earlier quoted context omitted.

That's on-chip SRAM, comparable to a GPU's L1 cache, of which it typically has megabytes . CSE systems also come with off-chip memory, comparable to a GPU's memory, but usually in the TB range.

If you want the titled 2500 tokens/second, you need to use the on-chip SRAM

What?

Of course they're using the on-chip SRAM, why wouldn't they?

This is a press release from Cerebras about a Cerebras chip, ... of course they are using a Cerebras chip!

Is that not obvious?

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#72

Post by rkomorn »

moralestapia wrote: post #70
ryao wrote: post #67

Earlier quoted context omitted.

The memory bandwidth for that is 150GB/sec. Inference speed is memory bandwidth bound, so that memory is useless for inference. Discrete GPUs will run circles around the CSE-3 at inference if they tried using the external DRAM.

Where do you get those 150GB/sec from? Here [1] they imply they can reach 1.2Tbps (allegedly, I know), and that's the previous generation ... 1: https://f.hubspotusercontent30.net/hubfs/8968533/Virtual%20B...

Doesn't 1.2Tbps / 8 = 150 GBps because 8b = 1B ?

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#73

Post by ryao »

impossiblefork wrote: post #14
ryao wrote: post #4

Earlier quoted context omitted.

Their CEO is a felon who plead guilty to accounting fraud: https://milled.com/theinformation/cerebras-ceos-past-felony-... Experienced investors will not touch them: https://www.nbclosangeles.com/news/business/money-report/cer... I estimated last year that they can only produce about 300 chips per year and that is unlikely to change because there are far bigger customers for TSMC that are ahead of them in priority fo…

While the CEO stuff is a problem, I don't think the other stuff matters. Per chip area WSE-3 is only a little bit more expensive than H200. While you may need several WSE-3s to load the model, if you have enough demand that you are running the WSE-3 at full speed you will not be using more area in the WSE-3. In fact, the WSE-3 may be more efficient, since it won't be loading and unloading things from large memories.…

I did the math last year to estimate how many wafers per year Nvidia had, and from my recollection it was >50,000. Cerebras with their ~300 per year is not able to handle the inference needs of the market. It does not help that all of their memory must be inside the wafer, which limits the amount of die area they have for actual logic. They have no prospect for growth unless TSMC decides to bless them or they switch to another foundation.

> While you may need several WSE-3s to load the model, if you have enough demand that you are running the WSE-3 at full speed you will not be using more area in the WSE-3.

You need ~20 wafers to run the Llama 4 Behemoth model on Cerebras hardware. This is close to a million mm^2. The Nvidia hardware that they used in their comparison should have less than 10,000 mm^2 die area, yet can run it fine thanks to the external DRAM. How is the CSE-3 not using more die area?

> In fact, the WSE-3 may be more efficient, since it won't be loading and unloading things from large memories.

This makes no sense to me. Inference software loads the model once and then uses it multiple times. This should be the same for both Nvidia and Cerebras.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#74

Post by moralestapia »

rkomorn wrote: post #72
moralestapia wrote: post #70

Earlier quoted context omitted.

Where do you get those 150GB/sec from? Here [1] they imply they can reach 1.2Tbps (allegedly, I know), and that's the previous generation ... 1: https://f.hubspotusercontent30.net/hubfs/8968533/Virtual%20B...

Doesn't 1.2Tbps / 8 = 150 GBps because 8b = 1B ?

That's ... right! Huh, missed that (assuming all units were written properly and mean what they mean).

Edit: yeah, double checked their site and everything. Dang, their IO is indeed "slow". They claim 1 microsecond latencies, but still, an H100 can move much more data than that.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#75

Post by ryao »

skryl wrote: post #59
ryao wrote: post #3

> At over 2,500 t/s, Cerebras has set a world record for LLM inference speed on the 400B parameter Llama 4 Maverick model, the largest and most powerful in the Llama 4 family. This is incorrect. The unreleased Llama 4 Behemoth is the largest and most powerful in the Llama 4 family. As for the speed record, it seems important to keep it in context. That comparison is only for performance on 1 query, but it is well kno…

Performance per watt is better than h100 and b200, performance per watt per $ is worse than B200, and it does fp8 just fine https://arxiv.org/pdf/2503.11698

Thanks for the correction. They are currently using FP16 for inference according to OpenRouter. I had thought that implied that they could not use FP8 given the pressure that they have to use as little memory as possible from being solely reliant on SRAM. I wonder why they opted to use FP16 instead of FP8.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#76

Post by ryao »

bubblethink wrote: post #12
ryao wrote: post #3

> At over 2,500 t/s, Cerebras has set a world record for LLM inference speed on the 400B parameter Llama 4 Maverick model, the largest and most powerful in the Llama 4 family. This is incorrect. The unreleased Llama 4 Behemoth is the largest and most powerful in the Llama 4 family. As for the speed record, it seems important to keep it in context. That comparison is only for performance on 1 query, but it is well kno…

>Each one costs ~$2 million, so that is $40 million. Pricing for exotic hardware that is not manufactured at scale is quite meaningless. They are selling tokens over an API. The token pricing is competitive with other token APIs.

Last year, I took the time to read through public documents and estimated that their annual production was limited to ~300 wafers per year from TSMC. That is not Nvidia level scale, but it is scale.

There are many companies that sell tokens from an API and many more that need hardware to compute tokens. Cerebras posted a comparison of hardware options for these companies, so evaluating it as such is meaningful. It is perhaps less meaningful to the average person who cannot afford the barrier to entry to afford this hardware, but there are plenty of people curious what the options are for the companies that sell tokens through APIs, as those impact available capacity.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#77

Post by ryao »

addaon wrote: post #63
ryao wrote: post #3

> At over 2,500 t/s, Cerebras has set a world record for LLM inference speed on the 400B parameter Llama 4 Maverick model, the largest and most powerful in the Llama 4 family. This is incorrect. The unreleased Llama 4 Behemoth is the largest and most powerful in the Llama 4 family. As for the speed record, it seems important to keep it in context. That comparison is only for performance on 1 query, but it is well kno…

> SRAM scaling is dead I'm /way/ outside my expertise here, so possibly-silly question. My understanding (any of which can be wrong, please correct me!) is that (a) the memory used for LLMs is dominantly parameters, which are read-only during inference; (b) SRAM scaling may be dead, but NVM scaling doesn't seem to be; (c) NVM read bandwidth scales well locally, within an order of magnitude or two of SRAM bandwidth, f…

The processes used for logic chips, and the processes used for NVM are typically different. The only case I know of the industry combining them onto a single chip would be Texas Instruments’ MSP430 microcontrollers with FeRAM, but the quantities of FeRAM are incredibly small there and the process technology is ancient. It seems unlikely to me that the rest of the industry will combine the processes such that you can have both on a single wafer, but you would have better luck asking a chip designer.

That said, NVM often has a wear-out problem. This is a major disincentive for using it in place of SRAM, which is frequently written. Different types of NVM have different endurance limits, but if they did build such a chip, it is only a matter of time before it stops working.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#78

Post by impossiblefork »

ryao wrote: post #73
impossiblefork wrote: post #14

Earlier quoted context omitted.

While the CEO stuff is a problem, I don't think the other stuff matters. Per chip area WSE-3 is only a little bit more expensive than H200. While you may need several WSE-3s to load the model, if you have enough demand that you are running the WSE-3 at full speed you will not be using more area in the WSE-3. In fact, the WSE-3 may be more efficient, since it won't be loading and unloading things from large memories.…

I did the math last year to estimate how many wafers per year Nvidia had, and from my recollection it was >50,000. Cerebras with their ~300 per year is not able to handle the inference needs of the market. It does not help that all of their memory must be inside the wafer, which limits the amount of die area they have for actual logic. They have no prospect for growth unless TSMC decides to bless them or they switch…

Yes, on an ordinary GPU it loads the weights to GPU memory, but then these weights must be moved from GPU memory onto the chip. But on these the weights can presumably be kept on chip entirely-- that's basically their whole point, so with the Cerebras there's no need to ever move weights to the chip.

Of course these guys depend on getting chips, but so does everybody. I don't know how difficult it is, but all sorts of entities get TSMC 5nm. Maybe they'll get TSMC 3nm and 2nm later than NVIDIA, but it's also possible that they don't.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#79

Post by ryao »

attentive wrote: post #11
ryao wrote: post #3

> At over 2,500 t/s, Cerebras has set a world record for LLM inference speed on the 400B parameter Llama 4 Maverick model, the largest and most powerful in the Llama 4 family. This is incorrect. The unreleased Llama 4 Behemoth is the largest and most powerful in the Llama 4 family. As for the speed record, it seems important to keep it in context. That comparison is only for performance on 1 query, but it is well kno…

> Also, Cerebras is the company that not only was saying that their hardware was not useful for inference until some time last year, but even partnered with Qualcomm with the claim that Qualcomm’s accelerators had a 10x price performance improvement over their things Mistral says they run Le Chat on Cerebras

How is that related to the claim that Cerebras themselves made about their hardware’s price performance ratio?

https://www.cerebras.ai/press-release/cerebras-qualcomm-anno...

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#80

Post by ryao »

moralestapia wrote: post #70
ryao wrote: post #67

Earlier quoted context omitted.

The memory bandwidth for that is 150GB/sec. Inference speed is memory bandwidth bound, so that memory is useless for inference. Discrete GPUs will run circles around the CSE-3 at inference if they tried using the external DRAM.

Where do you get those 150GB/sec from? Here [1] they imply they can reach 1.2Tbps (allegedly, I know), and that's the previous generation ... 1: https://f.hubspotusercontent30.net/hubfs/8968533/Virtual%20B...

The other comment already clarified that 150GB/sec = 1.2Tbps. That said, the CSE-3 did not change this figure. It is buried in their specification sheets somewhere if you care to search for it. I did last year, which is how I know.
Post reply on HN