Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

cerebras.ai

Post reply on HN
Print view View original

31–40 of 87 posts

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#31

Post by londons_explore »

simonw wrote: post #20
londons_explore wrote: post #15

Earlier quoted context omitted.

The "AI overview" in google search seems to be a similar speed, and the resulting text of similar quality.

I wonder which of their models they use. Might even be Gemini 1.5 Flash 8B which is VERY quick. I just tried that out with the same prompt and it's fast, but not as fast as Cerebras: https://static.simonwillison.net/static/2024/gemini-flash-8b...

I suspect it is its own model. Running it on 10B+ user queries per day you're gonna want to optimize everything you can about it - so you'd want something really optimized to the exact problem rather than using a general purpose model with careful prompting.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#32

Post by bubblethink »

germanjoey wrote: post #25
odo1242 wrote: post #14

What made it so much faster based on just a software update?

They said in the announcement that they've implemented speculative decoding, so that might have a lot to do with it. A big question is what they're using as their draft model; there's ways to do it losslessly, but they could also choose to trade off accuracy for a bigger increase in speed. It seems they also support only a very short sequence length. (1k tokens)

Speculative decoding does not trade off accuracy. You reject the speculated tokens if the original model does not accept them, kind of like branch prediction. All these providers and third parties benchmark each other's solutions, so if there is a drop in accuracy, someone will report it. Their sequence length is 8k.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#33

Post by campers »

odo1242 wrote: post #14

What made it so much faster based on just a software update?

  The first implementation of inference on the Wafer Scale Engine and utilized only a fraction of its peak bandwidth, compute, and IO capacity. Today’s release is the culmination of numerous software, hardware, and ML improvements we made to our stack to greatly improve the utilization and real-world performance of Cerebras Inference.
 
  We’ve re-written or optimized the most critical kernels such as MatMul, reduce/broadcast, element wise ops, and activations. Wafer IO has been streamlined to run asynchronously from compute. This release also implements speculative decoding, a widely used technique that uses a small model and large model in tandem to generate answers faster.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#34

Post by bestest »

selcuka wrote: post #9
anonzzzies wrote: post #3

Demo, API?

Demo: https://inference.cerebras.ai/ API: https://cloud.cerebras.ai/

I filled out a lengthy prompt in the demo. submitted it. an auth window pops up. I don't want to login. I want the demo. such a repulsive approach.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#35

Post by regularfry »

OhNoNotAgain_99 wrote: post #28
vineyardmike wrote: post #13

Earlier quoted context omitted.

If you're using an LLM as a compressed version of a search index, you'll be constantly fighting hallucinations. Respectfully, you're not thinking big-picture enough. There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time.…

To be honest most LLM's are reasonable at coding, they're not great. Sure they can code small stuff. But the can't refactor large software projects, or upgrade them.

Upgrading large java projects is exactly what AWS want you to believe their tooling can do, but the ergonomics aren't great.

I think most of the capability problems with coding agents aren't the AI itself, it's that we haven't cracked how to let them interact with the codebase effectively yet. When I refactor something, I'm not doing it all at once, it's a step by step process. None of the individual steps are that complicated. Translating that over to an agent feels like we just haven't got the right harness yet.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#36

Post by maz1b »

Cerebras really has impressed me with their technicality and their approach in the modern LLM era. I hope they do well, as I've heard they are en-route to IPO. It will be interesting to see if they can make a dent vs NVIDIA and other players in this space.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#37

Post by ben_w »

yunohn wrote: post #27
ben_w wrote: post #24

Earlier quoted context omitted.

By comparison with reality. The initial LLMs had "reality" be "a training set of text", when ChatGPT came out everyone rapidly expanded into RLFH (reinforcement learning from human feedback), and now there's vision and text models the training and feedback is grounded on a much broader aspect of reality than just text.

Could you link to a paper or working POC that shows how this “turtles all the way down“ solution works?

I don't understand your question.

This isn't turtles all the way down, it's grounded in real world data, and increasingly large varieties of it.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#38

Post by Arn_Thor »

obviyus wrote: post #18

Wonder if they'll eventually release Whisper support. Groq has been great for transcribing 1hr+ calls at a significnatly lower price compared to OpenAI ($0.36/hr vs. $0.04/hr).

Whisper runs so well locally on any hardware I’ve thrown at it, why run it in the cloud?

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#39

Post by swores »

Arn_Thor wrote: post #38
obviyus wrote: post #18

Wonder if they'll eventually release Whisper support. Groq has been great for transcribing 1hr+ calls at a significnatly lower price compared to OpenAI ($0.36/hr vs. $0.04/hr).

Whisper runs so well locally on any hardware I’ve thrown at it, why run it in the cloud?

Does it run well on CPU? I've used it locally but only with my high end (consumer/gaming) GPU, and haven't got round to finding out how it does on weaker machines.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#40

Post by croes »

simonw wrote: post #8

It turns out someone has written a plugin for my LLM CLI tool already: https://github.com/irthomasthomas/llm-cerebras You need an API key - I got one from https://cloud.cerebras.ai/ but I'm not sure if there's a waiting list at the moment - then you can do this: pipx install llm # or brew install llm or uv tool install llm llm install llm-cerebras llm keys set cerebras # paste key here Then you can run lightning fast…

It has a waiting list
Post reply on HN