I wonder if there is a token/watt metric. Afaiu cerebras uses plenty of power/cooling.
Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
41–50 of 87 posts
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#42Damn, that's some impressive speeds. At that rate it doesn't matter if the first try resulted in an unwanted answer, you'll be able to run once or twice more in a fast succession. I hope their hardware stays relevant as this field continues to evolve
The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.
Exactly, validating and rewriting the prompt are the real time consuming tasks.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#43Earlier quoted context omitted.
The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.
If you're using an LLM as a compressed version of a search index, you'll be constantly fighting hallucinations. Respectfully, you're not thinking big-picture enough. There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time.…
> There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time.
Well now the compiler is the bottleneck isn't it? And you would still need human check for bugs that aren't caught by the compiler.
Still nice to have inference speed improvements tho.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#44Earlier quoted context omitted.
Demo: https://inference.cerebras.ai/ API: https://cloud.cerebras.ai/
I filled out a lengthy prompt in the demo. submitted it. an auth window pops up. I don't want to login. I want the demo. such a repulsive approach.
chill with the emotionally charged words. their hardware, their rules. if this upsets you you will not have a good time on the modern internet.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#45Wonder if they'll eventually release Whisper support. Groq has been great for transcribing 1hr+ calls at a significnatly lower price compared to OpenAI ($0.36/hr vs. $0.04/hr).
Whisper runs so well locally on any hardware I’ve thrown at it, why run it in the cloud?
That's pretty much exactly how I started. Ran whisper.cpp locally for a while on a 3070Ti. It worked quite well when n=1.
For our use case, we may get 1 audio file at a time, we may get 10. Of course queuing them is possible but we decided to prioritize speed & reliability over self hosting.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#46Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#47Cerebras really has impressed me with their technicality and their approach in the modern LLM era. I hope they do well, as I've heard they are en-route to IPO. It will be interesting to see if they can make a dent vs NVIDIA and other players in this space.
Apparently so. You can also buy in via various PE outfits before IPO, if you so desire. I did.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#48Earlier quoted context omitted.
The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.
> The biggest time sink for me is validating answers so not sure I agree on that take. But you're assuming that it'll always ne validated by humans. I'd imagine that most validation (and subsequent processing, especially going forward) will be done on machines.
And who validates the validation?
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#49Earlier quoted context omitted.
How does the next machine/LLM know what’s valid or not? I don’t really understand the idea behind layers of hallucinating LLMs.
By comparison with reality. The initial LLMs had "reality" be "a training set of text", when ChatGPT came out everyone rapidly expanded into RLFH (reinforcement learning from human feedback), and now there's vision and text models the training and feedback is grounded on a much broader aspect of reality than just text.
Given that there are more and more AI generated texts and pictures that ground will be pretty unreliable.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#50Earlier quoted context omitted.
Could you link to a paper or working POC that shows how this “turtles all the way down“ solution works?
I don't understand your question. This isn't turtles all the way down, it's grounded in real world data, and increasingly large varieties of it.
How does the AI know it’s reality and not a fake image or text fed to the system?