So what is inference?
Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
51–60 of 87 posts
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#52Wonder if they'll eventually release Whisper support. Groq has been great for transcribing 1hr+ calls at a significnatly lower price compared to OpenAI ($0.36/hr vs. $0.04/hr).
https://Lemonfox.ai is another alternative to OpenAI's Whisper API if you need support for word-level timestamps and diarization.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#53Wow, software is hard! Imagine an entire company working to build an insanely huge and expensive wafer scale chip and your super smart and highly motivated machine learning engineers get 1/3 of peak performance on their first attempt. When people say NVIDIA has no moat I'm going to remember this - partly because it does show that they do, and partly because it shows that with time the moat can probably be crossed...
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#54Earlier quoted context omitted.
> The biggest time sink for me is validating answers so not sure I agree on that take. But you're assuming that it'll always ne validated by humans. I'd imagine that most validation (and subsequent processing, especially going forward) will be done on machines.
And who validates the validation?
the compiler/interpreter are assumed to work in this scenario.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#55I wonder if there is a token/watt metric. Afaiu cerebras uses plenty of power/cooling.
I found this on their product page, though just for peak power:
> At 16 RU, and peak sustained system power of 23kW, the CS-3 packs the performance of a room full of servers into a single unit the size of a dorm room mini-fridge.
It's pretty impressive looking hardware.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#56So what is inference?
Inference just means using the model, rather than training it.
As far as I know Nvidia still has a monopoly on the training part.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#57When Meta releases the quantized 70B it will give another > 2X speedup with similar accuracy: https://ai.meta.com/blog/meta-llama-quantized-lightweight-mo...
You don't need quantization aware training on larger models. 4 bit 70b and 405b models exhibit close to zero degradation in output with post training quantization[1][2]. [1]: https://arxiv.org/pdf/2409.11055v1 [2]: https://lmarena.ai/
I wonder why that is? because they are trained with dropout?
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#58[deleted]
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#59Earlier quoted context omitted.
If you're using an LLM as a compressed version of a search index, you'll be constantly fighting hallucinations. Respectfully, you're not thinking big-picture enough. There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time.…
> There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time. Well now the compiler is the bottleneck isn't it? And you would still need human check for bugs that aren't caught by the compiler. Still nice to have inference spe…
Something will always be the bottleneck, and it probably won’t be the speed of electrons for a while ;)
Some compilers (go) are faster than others (javac) and some languages are interpreted and can only be checked through tests. Moving the bottleneck from AI code gen step to the same bottleneck as a person seems like a win.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#60Earlier quoted context omitted.
The novelty of the inexplicable bitnet obsession has worn off I think.
We have yet to see a large model trained using it, haven't we?
Bitnet models are just another piece in the ocean of techniques where there may possibly be alpha at large parameter counts... but no one will know until a massive investment is made, and that investment hasn't happened because the people with resources have much surer things to invest in.
There's this insufferable crowd of people who just keep going on and on about it like it's some magic bullet that will let them run 405B on their home PC but if it was so simple it's not like the 5 or so companies in the world putting out frontier models need little Timmy 3090 to tell them about the technique: we don't need it shoehorned into every single release.