Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras Inference: AI at Instant Speed

cerebras.ai

Post reply on HN
Print view View original

31–40 of 75 posts

Re: Cerebras Inference: AI at Instant Speed

#31

Post by fittom »

smusamashah wrote: post #19

Can really fast inference (e.g. 1M tok/sec) make LLMs more intelligent? I am imagining you could run multiple agents and can choose and discard outputs using other LLMs simultaneously. Will the output look more like a real thought process? Or will it remain just same?

It is mentioned in the post:

> Traditional LLMs output everything they think immediately, without stopping to consider the best possible answer. New techniques like scaffolding, on the other hand, function like a thoughtful agent who explores different possible solutions before deciding. This “thinking before speaking” approach provides over 10x performance on demanding tasks like code generation, fundamentally boosting the intelligence of AI models without additional training.

Re: Cerebras Inference: AI at Instant Speed

#32

Post by bigcat12345678 »

sanxiyn wrote: post #28
bigcat12345678 wrote: post #17

Earlier quoted context omitted.

This question is a bit out of context. Cerebras is a startup producing innovative AI chips. Their chips are super cool, and I personally believe Cerebras is ahead of the industry and is on the right technical path. As a matter of fact, Cerebras started with HPC chips. Then pivoted to AI like everyone else. They are still deep in the trench for survival. Given that, they have very little software prowess compared to A…

As OpenAI themselves admits, structured output feature in question was developed in open source world first with zero funding.

This point is moot.

The point remain that Cerebras is not in a position to focus on structured output or prompt caching.

Re: Cerebras Inference: AI at Instant Speed

#33

Post by throwup238 »

jedberg wrote: post #22
sanxiyn wrote: post #9

Earlier quoted context omitted.

Doubtful. SRAM is not cheap, and this is entirely about SRAM vs HBM.

They list the price in this press release. So either they're taking a big loss or they're doing it cheaper per token.

It wouldn't be the first time a manufacturer ignored capital amortization to post better numbers.

Re: Cerebras Inference: AI at Instant Speed

#34

Post by ChainOfFools »

throwawaymaths wrote: post #11
itishappy wrote: post #5

Earlier quoted context omitted.

Cerebras' Wafer Scale Engine is the opposite of small and cheap. https://cerebras.ai/product-chip/

That's totally nuts. How do they deal with the silicon warping around disabled cores or dark silicon? How long of hard running does it take before the chip gets fatally damaged and needs to be replaced in their system? Word on the street is that h100s fail surprisingly often, this can't be better

edit: I should have just checked their website instead of guessing. Apparently WSE has significant fabrication challenges, which makes what Cerebras has accomplished all the more impressive. But it is still surprising that no one else has attempted this in the HPC field.

I had guessed that Cerebras had made some trade-offs in process in order to make it work at scale, but then they aren't actually building these devices at scale (yet).

Re: Cerebras Inference: AI at Instant Speed

#35

Post by nomel »

throwawaymaths wrote: post #11
itishappy wrote: post #5

Earlier quoted context omitted.

Cerebras' Wafer Scale Engine is the opposite of small and cheap. https://cerebras.ai/product-chip/

That's totally nuts. How do they deal with the silicon warping around disabled cores or dark silicon? How long of hard running does it take before the chip gets fatally damaged and needs to be replaced in their system? Word on the street is that h100s fail surprisingly often, this can't be better

Is this all mostly a heat spreader efficiency requirement?

Re: Cerebras Inference: AI at Instant Speed

#36

Post by sanxiyn »

jedberg wrote: post #22
sanxiyn wrote: post #9

Earlier quoted context omitted.

Doubtful. SRAM is not cheap, and this is entirely about SRAM vs HBM.

They list the price in this press release. So either they're taking a big loss or they're doing it cheaper per token.

Not even necessarily a loss let alone big, since their comparison includes margin.

Re: Cerebras Inference: AI at Instant Speed

#37

Post by johntb86 »

ChainOfFools wrote: post #34
throwawaymaths wrote: post #11

Earlier quoted context omitted.

That's totally nuts. How do they deal with the silicon warping around disabled cores or dark silicon? How long of hard running does it take before the chip gets fatally damaged and needs to be replaced in their system? Word on the street is that h100s fail surprisingly often, this can't be better

edit: I should have just checked their website instead of guessing. Apparently WSE has significant fabrication challenges, which makes what Cerebras has accomplished all the more impressive. But it is still surprising that no one else has attempted this in the HPC field. I had guessed that Cerebras had made some trade-offs in process in order to make it work at scale, but then they aren't actually building these devi…

They're using TSMC 5-nm for WSE-3: https://spectrum.ieee.org/cerebras-chip-cs3

Re: Cerebras Inference: AI at Instant Speed

#38

Post by sanxiyn »

ChainOfFools wrote: post #34
throwawaymaths wrote: post #11

Earlier quoted context omitted.

That's totally nuts. How do they deal with the silicon warping around disabled cores or dark silicon? How long of hard running does it take before the chip gets fatally damaged and needs to be replaced in their system? Word on the street is that h100s fail surprisingly often, this can't be better

edit: I should have just checked their website instead of guessing. Apparently WSE has significant fabrication challenges, which makes what Cerebras has accomplished all the more impressive. But it is still surprising that no one else has attempted this in the HPC field. I had guessed that Cerebras had made some trade-offs in process in order to make it work at scale, but then they aren't actually building these devi…

Cerebras is known to use TSMC, so your speculation about boutique fab is incorrect.

https://cerebras.ai/press-release/cerebras-systems-smashes-t...

Re: Cerebras Inference: AI at Instant Speed

#39

Post by ansk »

Is batched inference for LLMs memory bound? My understanding is that sufficiently large batched matmuls will be compute bound and flash attention has mostly removed the memory bottleneck in the attention computation. If so, the value proposition here -- as well as with other memorymaxxing startups like Groq -- is primarily on the latency side of things. Though my personal impression is that latency isn't really a huge issue right now, especially for text. Even OpenAI's voice models are (purportedly) able to be served with a latency which is a low multiple of network latency, and I expect there is room for improvement here as this is essentially the first generation of real-time voice LLMs.

Re: Cerebras Inference: AI at Instant Speed

#40

Post by mikewarot »

>Cerebras is the only platform to enable instant responses at a blistering 450 tokens/sec. All this is achieved using native 16-bit weights for the model, ensuring the highest accuracy responses.

As near as I can tell, from the model card[1], the majority of the math for this model is 4096x4096 multiply-accumulates. So, there should be 70b/16m about 4000 of these in the Llama3-70B model.

A 16x16 multiplier is about 9000 transistors, according to a quick google. 4096^2 should thus be about 150 billion transistors, if you include the bias values. There are plenty of transistors on this chip to have many of them operating in parallel.

According to [2], a switching transition in the 7nM process node, is about 0.025 femtoJoule (10^-15 watt seconds) per transistor. At a clock rate of 1 Ghz, that's about 25 nanowatt/transistor. Scaling that at 50% transitions(a 50/50 chance any given gate in the MAC will flip), gets you about 2kW for each 4096^2 MAC running at 1 Ghz.

There are enough transistors, and enough RAM on the wafer to fit the entire model. Even if they have a single 4096^2 MAC array, a clock rate of 1 ghz should result in a total time of 4 uSec/token, or 250,000 tokens/second.

[1] https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md

[2] https://mpedram.com/Papers/7nm-finfet-libraries-tcasII.pdf

Post reply on HN