Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

cerebras.ai

Post reply on HN
Print view View original

61–70 of 100 posts

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#61

Post by subscribed »

aktuel wrote: post #48
subscribed wrote: post #45

Earlier quoted context omitted.

LOL, all fintech are using or entering the "cloud" very heavily. Cloud is here for long enough that claiming it's insecure shows only the immense ignorance.

https://www.bleepingcomputer.com/news/security/oracle-custom... Just one of the later examples of a very long list of cloud data breaches affecting millions of users. But hey who cares as long as it does not affect your own bottom line.

This has affected login data and yeah, it's famously oracle.

Any fintech (and these can afford smart people) is building with defense in depth, encrypting everything with their own keys, using ephemeral credentials (eg issued by hashicorp vault), etc, etc.

You're seemingly applying your own experience with cloud-based storage, like Dropbox, to the enterprise cloud-based infrastructure.

I don't feel like I should spend any time laying out my professional experience with these environments, I guess you could just skim through one of the books and watch a couple hours long video explaining layers of the leading "cloud" offerings.

And yes, eventually the breach will happen. Like it happens on premise all the time. 2014 Sony and 2020 Solar Winds are good examples.

Let's agree to disagree, I really don't want to spend any more time on this, I know how a good solution (passing multiple audits and pentests) looks like, you however have your opinion. I'm not going to fight you :)

Take care!

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#62

Post by snypher »

subscribed wrote: post #45
aktuel wrote: post #43

Earlier quoted context omitted.

Some quant-heads endorsing the latest fad doesn't prove anything. Also they don't care if chinese hackers are vacuuming data cause ballstreet doesn't care about sustainability. But I grant you that secure and trust are just words that don't mean anything anymore anyhow.

LOL, all fintech are using or entering the "cloud" very heavily. Cloud is here for long enough that claiming it's insecure shows only the immense ignorance.

>Cloud is here for long enough that claiming it's insecure shows only the immense ignorance

Such a bizarre interpretation considering we still use SMS

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#63

Post by addaon »

ryao wrote: post #3

> At over 2,500 t/s, Cerebras has set a world record for LLM inference speed on the 400B parameter Llama 4 Maverick model, the largest and most powerful in the Llama 4 family. This is incorrect. The unreleased Llama 4 Behemoth is the largest and most powerful in the Llama 4 family. As for the speed record, it seems important to keep it in context. That comparison is only for performance on 1 query, but it is well kno…

> SRAM scaling is dead

I'm /way/ outside my expertise here, so possibly-silly question. My understanding (any of which can be wrong, please correct me!) is that (a) the memory used for LLMs is dominantly parameters, which are read-only during inference; (b) SRAM scaling may be dead, but NVM scaling doesn't seem to be; (c) NVM read bandwidth scales well locally, within an order of magnitude or two of SRAM bandwidth, for wide reads; (d) although NVM isn't currently on leading-edge processes, market forces are generally pushing NVM to smaller and smaller processes for the usual cost/density/performance reasons.

Assuming that cluster of assumptions is true, does that suggest that there's a time down the road where something like a chip-scale-integrated inference chip using NVM for parameter storage solves?

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#64

Post by [deleted] »

threeseed wrote: post #41
arisAlexis wrote: post #27

Earlier quoted context omitted.

Openai wanted to buy them. G42 the largest player in middle east owne a big chunk. You are simply wrong about big investors not touching them but my guess is they will be bought soon by Meta or Apple.

> Apple I can't imagine Apple being interested. Their priority is figuring out how to optimise Apple Silicon for LLM inference so it can be used in laptops, phones and data centres.

[deleted]

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#65

Post by bigyabai »

threeseed wrote: post #41
arisAlexis wrote: post #27

Earlier quoted context omitted.

Openai wanted to buy them. G42 the largest player in middle east owne a big chunk. You are simply wrong about big investors not touching them but my guess is they will be bought soon by Meta or Apple.

> Apple I can't imagine Apple being interested. Their priority is figuring out how to optimise Apple Silicon for LLM inference so it can be used in laptops, phones and data centres.

I can only imagine Apple being interested. Their NPU hardware is slower than Qualcomm's, their GPUs have been lagging behind Nvidia in all fields since launch, and they refuse to work with any industry leaders to ship a COTS solution. They don't have many options left on the table, "figuring out how to optimize Apple Silicon" has been the plan for 6 years now and no CUDA-killers have come up out of the woodworks since then.

Either Apple entirely forfeits AI to the businesses capable of supplying it, or they change their tactic and do what Apple does best; grossly overpay for a moonshot startup that promises "X for the iPhone". I don't know if that implicates Cerebras, but clearly Apple didn't retain the requisite talent to compete for commercial AI inference capacity.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#66

Post by acchow »

moralestapia wrote: post #60
acchow wrote: post #24

Earlier quoted context omitted.

CSE-3 chip has 44GB, which can hold 22B parameters in FP16. 400B parameters would need 18 chips. Then you need a bit more ram for other stuff

That's on-chip SRAM, comparable to a GPU's L1 cache, of which it typically has megabytes . CSE systems also come with off-chip memory, comparable to a GPU's memory, but usually in the TB range.

If you want the titled 2500 tokens/second, you need to use the on-chip SRAM

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#67

Post by ryao »

moralestapia wrote: post #60
acchow wrote: post #24

Earlier quoted context omitted.

CSE-3 chip has 44GB, which can hold 22B parameters in FP16. 400B parameters would need 18 chips. Then you need a bit more ram for other stuff

That's on-chip SRAM, comparable to a GPU's L1 cache, of which it typically has megabytes . CSE systems also come with off-chip memory, comparable to a GPU's memory, but usually in the TB range.

The memory bandwidth for that is 150GB/sec. Inference speed is memory bandwidth bound, so that memory is useless for inference. Discrete GPUs will run circles around the CSE-3 at inference if they tried using the external DRAM.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#68

Post by lern_too_spel »

skryl wrote: post #59
ryao wrote: post #3

> At over 2,500 t/s, Cerebras has set a world record for LLM inference speed on the 400B parameter Llama 4 Maverick model, the largest and most powerful in the Llama 4 family. This is incorrect. The unreleased Llama 4 Behemoth is the largest and most powerful in the Llama 4 family. As for the speed record, it seems important to keep it in context. That comparison is only for performance on 1 query, but it is well kno…

Performance per watt is better than h100 and b200, performance per watt per $ is worse than B200, and it does fp8 just fine https://arxiv.org/pdf/2503.11698

Performance per watt per dollar is a useless metric as calculated. You can't spend more money on B200s to get more performance per watt.

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#69

Post by ryao »

bigyabai wrote: post #65
threeseed wrote: post #41

Earlier quoted context omitted.

> Apple I can't imagine Apple being interested. Their priority is figuring out how to optimise Apple Silicon for LLM inference so it can be used in laptops, phones and data centres.

I can only imagine Apple being interested. Their NPU hardware is slower than Qualcomm's, their GPUs have been lagging behind Nvidia in all fields since launch, and they refuse to work with any industry leaders to ship a COTS solution. They don't have many options left on the table, "figuring out how to optimize Apple Silicon" has been the plan for 6 years now and no CUDA-killers have come up out of the woodworks sinc…

Cerebras’ technology works by using an entire wafer as a chip and power draw is 23kW if I recall correctly. Their technology cannot be scaled down and only works when scaling up. They could not be more useless for Apple’s purposes. Acquiring them would only give them a bunch of chip design engineers that might or might not be able to make a decent NPU that uses DRAM.

That said, Apple has some talented people already and they likely just need to iterate to make their designs better. Bringing new people on board would just slow progress (see the mythical man month).

Re: Cerebras achieves 2,500T/s on Llama 4 Maverick (400B)

#70

Post by moralestapia »

ryao wrote: post #67
moralestapia wrote: post #60

Earlier quoted context omitted.

That's on-chip SRAM, comparable to a GPU's L1 cache, of which it typically has megabytes . CSE systems also come with off-chip memory, comparable to a GPU's memory, but usually in the TB range.

The memory bandwidth for that is 150GB/sec. Inference speed is memory bandwidth bound, so that memory is useless for inference. Discrete GPUs will run circles around the CSE-3 at inference if they tried using the external DRAM.

Where do you get those 150GB/sec from?

Here [1] they imply they can reach 1.2Tbps (allegedly, I know), and that's the previous generation ...

1: https://f.hubspotusercontent30.net/hubfs/8968533/Virtual%20B...

Post reply on HN