Demo, API?
Demo: https://inference.cerebras.ai/ API: https://cloud.cerebras.ai/
That's odd, attempting a prompt fails because auth isn't working.
Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
11–20 of 87 posts
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#12Earlier quoted context omitted.
The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.
> The biggest time sink for me is validating answers so not sure I agree on that take. But you're assuming that it'll always ne validated by humans. I'd imagine that most validation (and subsequent processing, especially going forward) will be done on machines.
If that is the way to get quality, sure.
Otherwise I feel that power consumption is the bigger issue than speed, though in this case they are interlinked.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#13Damn, that's some impressive speeds. At that rate it doesn't matter if the first try resulted in an unwanted answer, you'll be able to run once or twice more in a fast succession. I hope their hardware stays relevant as this field continues to evolve
The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.
If you're using an LLM as a compressed version of a search index, you'll be constantly fighting hallucinations. Respectfully, you're not thinking big-picture enough.
There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time.
There are efforts to enable LLMs to "think" by using Chain-of-thought, where the LLM writes out reasoning in a "proof" style list of steps. Sometimes, like with a person, they'd reach a dead-end logic wise. If you can run 3x faster, you can start to run the "thought chain" as more of a "tree" where the logic is critiqued and adapted, and where many different solutions can be tried. This can all happen in parallel (well, each sub-branch).
Then there are "agent" use cases, where an LLM has to take actions on its own in response to real-world situations. Speed really impacts user-perception of quality.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#14What made it so much faster based on just a software update?
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#15It turns out someone has written a plugin for my LLM CLI tool already: https://github.com/irthomasthomas/llm-cerebras You need an API key - I got one from https://cloud.cerebras.ai/ but I'm not sure if there's a waiting list at the moment - then you can do this: pipx install llm # or brew install llm or uv tool install llm llm install llm-cerebras llm keys set cerebras # paste key here Then you can run lightning fast…
The "AI overview" in google search seems to be a similar speed, and the resulting text of similar quality.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#16Could someone please bring Microsoft's Bitnet into the discussion and explain how its performance relates to this announcement, if at all? https://github.com/microsoft/BitNet "bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups ran…
The novelty of the inexplicable bitnet obsession has worn off I think.
We have yet to see a large model trained using it, haven't we?
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#17Earlier quoted context omitted.
The biggest time sink for me is validating answers so not sure I agree on that take. Fast iteration is a killer feature, for sure, but at this time I'd rather focus on quality for it to be worthwhile the effort.
If you're using an LLM as a compressed version of a search index, you'll be constantly fighting hallucinations. Respectfully, you're not thinking big-picture enough. There are LLMs today that are amazing at coding, and when you allow it to iterate (eg. respond to compiler errors), the quality is pretty impressive. If you can run an LLM 3x faster, you can enable a much bigger feedback loop in the same period of time.…
If the speed is used to get better quality with no more input from the user then sure, that is great. But that is not the only way to get better quality (though I agree that there are some low hanging fruit in the area).
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#18Wonder if they'll eventually release Whisper support. Groq has been great for transcribing 1hr+ calls at a significnatly lower price compared to OpenAI ($0.36/hr vs. $0.04/hr).
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#19Earlier quoted context omitted.
> The biggest time sink for me is validating answers so not sure I agree on that take. But you're assuming that it'll always ne validated by humans. I'd imagine that most validation (and subsequent processing, especially going forward) will be done on machines.
If that is the way to get quality, sure. Otherwise I feel that power consumption is the bigger issue than speed, though in this case they are interlinked.
Humans consume a lot of power and resources.
Re: Cerebras Inference now 3x faster: Llama3.1-70B breaks 2,100 tokens/s
#20It turns out someone has written a plugin for my LLM CLI tool already: https://github.com/irthomasthomas/llm-cerebras You need an API key - I got one from https://cloud.cerebras.ai/ but I'm not sure if there's a waiting list at the moment - then you can do this: pipx install llm # or brew install llm or uv tool install llm llm install llm-cerebras llm keys set cerebras # paste key here Then you can run lightning fast…
The "AI overview" in google search seems to be a similar speed, and the resulting text of similar quality.
I wonder which of their models they use. Might even be Gemini 1.5 Flash 8B which is VERY quick.
I just tried that out with the same prompt and it's fast, but not as fast as Cerebras: https://static.simonwillison.net/static/2024/gemini-flash-8b...