Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

51–60 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#51

Post by wmf »

mrtksn wrote: post #41

Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mobile devices. Or maybe have a dedicated storage, RAM and processing cores that goes as an USB-C add-on? A case with integrated battery and this chip? I'm dreaming of having LLMs on anything. Unlike the "bluetooth on everything" craze, this can be practical as…

I assume this is a million-dollar rack of custom chips so it's probably not coming to mobile any time soon.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#53

Post by ukuina »

CuriouslyC wrote: post #23

This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.

I wonder if Gemini Pro 1.5 will act as a forcing function to lower GPT4 pricing.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#54

Post by xnorswap »

Workaccount2 wrote: post #48
xnorswap wrote: post #38

Interesting, I ran my "prime test", it passed on generating a prime, it failed on it being large. > "Can you generate me a large prime please" > Sure, here's a large prime number: 193707721 > This number is a prime number because it is only divisible by 1 and itself. You can check its primality by trying to divide it by all the numbers up to its square root, and if none of them divide it evenly, then it is a prime nu…

I can't think of any value in asking LLM's to generate prime numbers. They are neural nets, not turing machines.

The value is showing how confidently is presents incorrect results.

Especially the lack of nuance or uncertainty in the language they use.

They extremely confidently present the incorrect information, and prime generation is interesting because it's information that isn't easy to spot as obviously incorrect to the user while being information that's possible to determine is wrong at small numbers and difficult to verify for large numbers.

It's my favourite test because it's a good demonstration of the lack of nuance or uncertainty in LLMs. They have no sense of how wrong the information they're giving out might be.

If they could give confidence intervals for any information then they could provide the context by how likely they think they might be correct, but they actually double-down on their incorrectness instead.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#57

Post by MuffinFlavored »

CuriouslyC wrote: post #23

This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.

What's the difference in your own words/opinion in quality between GPT-3.5 and GPT-4? For what usecases?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#58

Post by MuffinFlavored »

tome wrote: post #52
sorokod wrote: post #46

Not clear if it is due to Groq or to Mixtral, but confident hallucinations are there.

We run the open source models that everyone else has access to. What we're trying to show off is our low latency and high throughput, not the model itself.

But if the model is useless/full of hallucinations, why does the speed of its output matter?

"generate hallucinated results, faster"

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#59

Post by karpathy »

Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#60

Post by frozenport »

mrtksn wrote: post #41

Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mobile devices. Or maybe have a dedicated storage, RAM and processing cores that goes as an USB-C add-on? A case with integrated battery and this chip? I'm dreaming of having LLMs on anything. Unlike the "bluetooth on everything" craze, this can be practical as…

Yeah just offload the compute onto the cloud.
Post reply on HN