Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras Inference: AI at Instant Speed

cerebras.ai

Post reply on HN
Print view View original

71–75 of 75 posts

Re: Cerebras Inference: AI at Instant Speed

#71

Post by aurareturn »

aurareturn wrote: post #68
mikewarot wrote: post #40

>Cerebras is the only platform to enable instant responses at a blistering 450 tokens/sec. All this is achieved using native 16-bit weights for the model, ensuring the highest accuracy responses. As near as I can tell, from the model card[1], the majority of the math for this model is 4096x4096 multiply-accumulates. So, there should be 70b/16m about 4000 of these in the Llama3-70B model. A 16x16 multiplier is about 9…

>There are enough transistors, and enough RAM on the wafer to fit the entire model. Not the entire 70b fp16 model. It'd take 148GB of RAM to hold the entire model. Each Cerebras wafer chip has 44GB of SRAM. You need 4 of them chained together to hold the entire model.

Correction: 40GB of SRAM each. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

Re: Cerebras Inference: AI at Instant Speed

#72

Post by throwawaymaths »

ChainOfFools wrote: post #59
throwawaymaths wrote: post #53

Earlier quoted context omitted.

It shouldn't be surprising. It's hard as fuck, and we don't know if it's worth it yet (there's something to be said for "if your compute dies you send your remote hands to swap out a high four figure component" and not "decommission a high-six-figure node"

I hope I didn't make it sound like it was easy, at least I don't think I said that anywhere. It doesn't really matter how hard something is to do (short of it being trivially proven impossible), it matters whether there's a good enough chance that the payoff exceeds the cost. And actually there have been attempts to do it, I mentioned in an earlier version of my comment that Gene Amdahl had attemped to make WSE work…

What makes you think you couldn't do as well on something less radical?

IIRC cerebras' design was originally for HPC workloads, so even it may not necessarily be optimized for LLMs

Re: Cerebras Inference: AI at Instant Speed

#73

Post by jrnx »

aurareturn wrote: post #71
aurareturn wrote: post #68

Earlier quoted context omitted.

>There are enough transistors, and enough RAM on the wafer to fit the entire model. Not the entire 70b fp16 model. It'd take 148GB of RAM to hold the entire model. Each Cerebras wafer chip has 44GB of SRAM. You need 4 of them chained together to hold the entire model.

Correction: 40GB of SRAM each. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...

40 GB was the CS-2. CS-3 has 44 GB

https://cerebras.ai/blog/cerebras-cs3

Re: Cerebras Inference: AI at Instant Speed

#74

Post by alecco »

alecco wrote: post #70

> Thus to generate a 100 words a second requires moving the model 100 times per second – requiring vast amounts of memory bandwidth. It's actually worse for the majority of GPU implementations for large models. The matrices don't fit shared memory so the model is loaded many, many times to shared memory (as tiles). Also, unless you are using Hopper distributed shared memory, CTAs can't even share across them. It woul…

It seems they are doing fine-tuning and soon pre-training with their new cluster technology. They claim to be able to train Llama 3 70GB in 1 day (instead of 1 month). Impressive if true.

Re: Cerebras Inference: AI at Instant Speed

#75

Post by joshhug »

russ wrote: post #46

It’s insanely fast. Here’s an AI voice assistant I built that uses it: https://cerebras.vercel.app

That was interesting. I asked it to try to say something in another language, and she read it in a thick American accent. No surprise. Then I asked her to sing, and she said something like "asterisk in a robotic singing voice asterisk...", and then later explained that she's just text to speech. Ah, ok, that's about what I expected.

But then I asked her to integrate sin(x) * e^x and got this bizarre answer that started out as speech sounds but then degenerated into chaos. Out of curiosity, why and how did she end up generating samples that sounded rather unlike speech?

Here's a recording: https://youtu.be/wWhxF7ybiAc

FWIW, I can get this behavior pretty consistently if I chat with her a while about her voice capabilities and then go into a math question.

Post reply on HN