Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

cerebras.ai

Post reply on HN
Print view View original

21–30 of 87 posts

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#21

Post by yunohn »

jeswin wrote: post #10
tjoff wrote: post #7

Earlier quoted context omitted.

The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.

> The biggest time sink for me is validating answers so not sure I agree on that take. But you're assuming that it'll always ne validated by humans. I'd imagine that most validation (and subsequent processing, especially going forward) will be done on machines.

How does the next machine/LLM know what’s valid or not? I don’t really understand the idea behind layers of hallucinating LLMs.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#23

Post by anon291 »

odo1242 wrote: post #14

What made it so much faster based on just a software update?

Ex-cereberas engineer here. The chip is very powerful and there is no 'one way' to do things. Rearchitecting data flow, changing up data layout, etc can lead to significant performance improvements. That's just my informed speculation. There's likely more perf somewhere

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#24

Post by ben_w »

yunohn wrote: post #21
jeswin wrote: post #10

Earlier quoted context omitted.

> The biggest time sink for me is validating answers so not sure I agree on that take. But you're assuming that it'll always ne validated by humans. I'd imagine that most validation (and subsequent processing, especially going forward) will be done on machines.

How does the next machine/LLM know what’s valid or not? I don’t really understand the idea behind layers of hallucinating LLMs.

By comparison with reality. The initial LLMs had "reality" be "a training set of text", when ChatGPT came out everyone rapidly expanded into RLFH (reinforcement learning from human feedback), and now there's vision and text models the training and feedback is grounded on a much broader aspect of reality than just text.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#25

Post by germanjoey »

odo1242 wrote: post #14

What made it so much faster based on just a software update?

They said in the announcement that they've implemented speculative decoding, so that might have a lot to do with it.

A big question is what they're using as their draft model; there's ways to do it losslessly, but they could also choose to trade off accuracy for a bigger increase in speed.

It seems they also support only a very short sequence length. (1k tokens)

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#26

Post by YetAnotherNick »

GavCo wrote: post #22

When Meta releases the quantized 70B it will give another > 2X speedup with similar accuracy: https://ai.meta.com/blog/meta-llama-quantized-lightweight-mo...

You don't need quantization aware training on larger models. 4 bit 70b and 405b models exhibit close to zero degradation in output with post training quantization[1][2].

[1]: https://arxiv.org/pdf/2409.11055v1 [2]: https://lmarena.ai/

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#27

Post by yunohn »

ben_w wrote: post #24
yunohn wrote: post #21

Earlier quoted context omitted.

How does the next machine/LLM know what’s valid or not? I don’t really understand the idea behind layers of hallucinating LLMs.

By comparison with reality. The initial LLMs had "reality" be "a training set of text", when ChatGPT came out everyone rapidly expanded into RLFH (reinforcement learning from human feedback), and now there's vision and text models the training and feedback is grounded on a much broader aspect of reality than just text.

Could you link to a paper or working POC that shows how this “turtles all the way down“ solution works?

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#28

Post by OhNoNotAgain_99 »

vineyardmike wrote: post #13
tjoff wrote: post #7

Earlier quoted context omitted.

The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.

If you're using an LLM as a compressed version of a search index, you'll be constantly fighting hallucinations. Respectfully, you're not thinking big-picture enough. There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time.…

To be honest most LLM's are reasonable at coding, they're not great. Sure they can code small stuff. But the can't refactor large software projects, or upgrade them.

Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s

#30

Post by qwertox »

BoorishBears wrote: post #5
andrewstuart wrote: post #2

Could someone please bring Microsoft's Bitnet into the discussion and explain how its performance relates to this announcement, if at all? https://github.com/microsoft/BitNet "bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups ran…

The novelty of the inexplicable bitnet obsession has worn off I think.

IDK, they remind me of Sigma-Delta ADCs [0], which are single bit ADCs but used in high resolution scenarios.

I believe we'll get to hear more interesting things about Bitnet in the future.

[0] https://en.wikipedia.org/wiki/Delta-sigma_modulation

Post reply on HN