Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

cerebras.ai

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#81

Post by ryao »

moralestapia wrote: post #71
acchow wrote: post #66

Earlier quoted context omitted.

If you want the titled 2500 tokens/second, you need to use the on-chip SRAM

What? Of course they're using the on-chip SRAM, why wouldn't they? This is a press release from Cerebras about a Cerebras chip, ... of course they are using a Cerebras chip! Is that not obvious?

They also support external DRAM over their 150GB/sec system IO link. They call it MemoryX and talk about it on these blog posts:

https://www.cerebras.ai/blog/cerebras-cs-3-vs-nvidia-b200-20...

https://www.cerebras.ai/blog/announcing-the-cerebras-archite...

It is useless for inference, but it is great for training. It used to be more prominent on their website, but it is harder to find references to it now that they are mimicking Groq’s business model.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#82

Post by baq »

tiffanyh wrote: post #46
baq wrote: post #35

Earlier quoted context omitted.

It is if you want good results. I’ve been giving Gemini pro prompts for 200+ seconds multiple times per day this week and for such tasks I really like to make it double/triple check and sometimes give the results to Claude for review, too (and vice versa). Ideally I can just run the prompt 100x and have it pick the best solution later. That’s prohibitively expensive and a waste of time today.

How do you create a prompt for Gemini to spend 200 seconds and review multiple times. Is it as simple as stating in the prompt: Spend 200+ seconds and review multiple times

You give it a task from hell which the devil himself outsources, like ‘figure out how these fifty repositories of yaml blobs, jinja templates and code generating code generating hcl generating yaml interact to define the infrastructure, then add something to it with correct iams, then make a matching blob of yaml pipelines to work with that infrastructure’

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#83

Post by baq »

diggan wrote: post #36
baq wrote: post #35

Earlier quoted context omitted.

It is if you want good results. I’ve been giving Gemini pro prompts for 200+ seconds multiple times per day this week and for such tasks I really like to make it double/triple check and sometimes give the results to Claude for review, too (and vice versa). Ideally I can just run the prompt 100x and have it pick the best solution later. That’s prohibitively expensive and a waste of time today.

> That’s prohibitively expensive Assuming you experience is working within enterprise, you're then saying that cost is the biggest bottleneck currently? Also surprising to me that enterprises would use out-of-the-box models like that, I was expecting at least fine-tuned models be used most of the time, for very specific tasks/contexts, but maybe that's way optimistic.

Cost would be the biggest factor if price per token was the same but tokens were arriving 100x faster. (Not particularly unexpected I’d say.)

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#84

Post by nsteel »

timschmidt wrote: post #28
nsteel wrote: post #26

Earlier quoted context omitted.

Assume you meant Intel, rather than AMD?

https://www.amd.com/en/products/processors/technologies/3d-v... and future developments.

Yes, and it's TSMC enabling this. Lots of TSMC's customers going this route, not just AMD. Seemed odd to call out AMD as if they've got any special sauce here.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#85

Post by addaon »

ryao wrote: post #77
addaon wrote: post #63

Earlier quoted context omitted.

> SRAM scaling is dead I'm /way/ outside my expertise here, so possibly-silly question. My understanding (any of which can be wrong, please correct me!) is that (a) the memory used for LLMs is dominantly parameters, which are read-only during inference; (b) SRAM scaling may be dead, but NVM scaling doesn't seem to be; (c) NVM read bandwidth scales well locally, within an order of magnitude or two of SRAM bandwidth, f…

The processes used for logic chips, and the processes used for NVM are typically different. The only case I know of the industry combining them onto a single chip would be Texas Instruments’ MSP430 microcontrollers with FeRAM, but the quantities of FeRAM are incredibly small there and the process technology is ancient. It seems unlikely to me that the rest of the industry will combine the processes such that you can…

> The only case I know of the industry combining them onto a single chip would be Texas Instruments’ MSP430 microcontrollers with FeRAM

Every microcontroller with on-chip NVM would count. Down to 45 nm, this is mostly Flash, with the exception of the MSP430's FeRAM. Below that... we have TI pushing Flash, ST pushing PCM, NXP pushing MRAM, and Infineon pushing (TSMC's) RRAM. All on processes in the 22 nm (planar) range, either today or in the near future.

> This is a major disincentive for using it in place of SRAM, which is frequently written.

But isn't parameter memory written once per model update, for silicon used for inferencing on a specific model? Even with daily writes the typical 10k - 1M allowable writes for most of the technologies above would last decades.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#86

Post by timschmidt »

nsteel wrote: post #84
timschmidt wrote: post #28

Earlier quoted context omitted.

https://www.amd.com/en/products/processors/technologies/3d-v... and future developments.

Yes, and it's TSMC enabling this. Lots of TSMC's customers going this route, not just AMD. Seemed odd to call out AMD as if they've got any special sauce here.

My choices can seem odd to you, that's fine. Have a nice day!

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#87

Post by ryao »

impossiblefork wrote: post #78
ryao wrote: post #73

Earlier quoted context omitted.

I did the math last year to estimate how many wafers per year Nvidia had, and from my recollection it was >50,000. Cerebras with their ~300 per year is not able to handle the inference needs of the market. It does not help that all of their memory must be inside the wafer, which limits the amount of die area they have for actual logic. They have no prospect for growth unless TSMC decides to bless them or they switch…

Yes, on an ordinary GPU it loads the weights to GPU memory, but then these weights must be moved from GPU memory onto the chip. But on these the weights can presumably be kept on chip entirely-- that's basically their whole point, so with the Cerebras there's no need to ever move weights to the chip. Of course these guys depend on getting chips, but so does everybody. I don't know how difficult it is, but all sorts o…

The CSE-3 is divided into 900,000 PEs, which each have only 48kB of RAM:

https://hc2024.hotchips.org/assets/program/conference/day2/7...

Similarly, the SMs in Blackwell have up to 228kB of RAM:

https://docs.nvidia.com/cuda/archive/12.8.0/pdf/Blackwell_Tu...

If you need anything else, you need to load it from elsewhere. In the CSE-3, that would be from other PEs. In Blackwell, that would be from on package DRAM. Idle time in Blackwell be mitigated by parallelism, since each SM has SRAM for multiple kernels to run in parallel. I believe the CSE-3 is quick enough that they do not need that trick.

The other guy said “you will not be using more area in the WSE-3”. I do not see this die area efficiency. You need many full wafers (around 20 with Llama 4 Maverick) to do the same thing with the CSE-3 that can be done with a fraction of a wafer with Blackwell. Even if you include the DRAM’s die area, Nvidia’s hardware is still orders of magnitude more efficient in terms of die area.

The only advantage Cerebras has as far as I can see is that they are fast on single queries, but they do not dare advertise figures for their total throughput, while Nvidia will happily advertise those. If they were better than Nvidia at throughput numbers, Cerebras would advertise them, since that is what matters for having mass market appeal, yet they avoid publishing those figures. That is likely because in reality, they are not competitive in throughput.

To give an example of Nvidia advertising throughput numbers:

> In a 1-megawatt AI factory, NVIDIA Hopper generates 180,000 tokens per second (TPS) at max volume, or 225 TPS for one user at the fastest.

https://blogs.nvidia.com/blog/ai-factory-inference-optimizat...

Cerebras strikes me as being like Bugatti, which designs cars that go from start to finish very fast at a price that could buy dozens of conventional vehicles, while Nvidia strikes me as being like Toyota, which designs far lower vehicles, but can manufacture them in a volume that is able to handle a large amount of the world’s demand for transport. Bugatti can make enough vehicles to bring a significant proportion of the world from A to B regularly, while Toyota can. Similarly, Cerebras cannot make enough chips to handle any significant proportion of the world’s demand for inference, while Nvidia can.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#88

Post by impossiblefork »

ryao wrote: post #87
impossiblefork wrote: post #78

Earlier quoted context omitted.

Yes, on an ordinary GPU it loads the weights to GPU memory, but then these weights must be moved from GPU memory onto the chip. But on these the weights can presumably be kept on chip entirely-- that's basically their whole point, so with the Cerebras there's no need to ever move weights to the chip. Of course these guys depend on getting chips, but so does everybody. I don't know how difficult it is, but all sorts o…

The CSE-3 is divided into 900,000 PEs, which each have only 48kB of RAM: https://hc2024.hotchips.org/assets/program/conference/day2/7... Similarly, the SMs in Blackwell have up to 228kB of RAM: https://docs.nvidia.com/cuda/archive/12.8.0/pdf/Blackwell_Tu... If you need anything else, you need to load it from elsewhere. In the CSE-3, that would be from other PEs. In Blackwell, that would be from on package DRAM. Idle…

I don't really see how NVIDIA shipping so many chips matters. If more people want Cerebras chips they will presumably be manufactured.

I agree that Cerebras manufacture I don't really see how that matters though. I don't see how core counts matter, but I assume that Cerebras is some kind of giant VLIW-y thing where you can give different instructions to different subprocessors.

I imagine that the model weights would be stored in little bits on each processor and that it does some calculation and hands it on.

Then you never need to load the the weights, the only thing you're passing around is activations with them going from wafer 1, to wafer 2, etc. to wafer 20. When this is running at full speed, I believe that this can be very efficient, better than a small GPU like those made by NVIDIA.

Yes, a lot of the area will be on-chip memory/SRAM, but a lot of it will also be logic and that logic will be computing things instead of being used to move things from RAM to on-chip memory.

I don't have any deep knowledge of this system, really, nothing beyond what I've explained here, but I believe that Mistral are using these systems because they're completely superb and superior to GPUs for their purposes, and they will made a carefully weighed decision based on actual performance and actual cost.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#89

Post by latchkey »

ryao wrote: post #76
bubblethink wrote: post #12

Earlier quoted context omitted.

>Each one costs ~$2 million, so that is $40 million. Pricing for exotic hardware that is not manufactured at scale is quite meaningless. They are selling tokens over an API. The token pricing is competitive with other token APIs.

Last year, I took the time to read through public documents and estimated that their annual production was limited to ~300 wafers per year from TSMC. That is not Nvidia level scale, but it is scale. There are many companies that sell tokens from an API and many more that need hardware to compute tokens. Cerebras posted a comparison of hardware options for these companies, so evaluating it as such is meaningful. It is…

> There are many companies that sell tokens from an API

I was just at Dell Tech World and they proudly displayed a slide during the CTO keynote that said:

"Cost per token decreased 4 orders of magnitude"

Personally speaking, not a business I'd want to get into.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#90

Post by latchkey »

timschmidt wrote: post #10
ryao wrote: post #3

> At over 2,500 t/s, Cerebras has set a world record for LLM inference speed on the 400B parameter Llama 4 Maverick model, the largest and most powerful in the Llama 4 family. This is incorrect. The unreleased Llama 4 Behemoth is the largest and most powerful in the Llama 4 family. As for the speed record, it seems important to keep it in context. That comparison is only for performance on 1 query, but it is well kno…

> I have trouble getting excited about Cerebras. SRAM scaling is dead, so short of figuring out how to 3D stack their wafer scale chips AMD and TSMC are stacking SRAM on the chip scale. I imagine they could accomplish it at the wafer scale. It'll be neat if we can get hundreds of layers in time, like flash. Your analysis seems spot on to me.

More on the CPU side than the GPU side. GPU is still dominated by HBM.
Post reply on HN