Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
[deleted]
Groq runs Mixtral 8x7B-32k with 500 T/s
61–70 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#62Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.
The thing is that tokens aren't an apples to apples metric.... Stupid tokens are a lot faster than clever tokens. I'd rather see token cleverness improving exponentially....
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#63Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mobile devices. Or maybe have a dedicated storage, RAM and processing cores that goes as an USB-C add-on? A case with integrated battery and this chip? I'm dreaming of having LLMs on anything. Unlike the "bluetooth on everything" craze, this can be practical as…
I assume this is a million-dollar rack of custom chips so it's probably not coming to mobile any time soon.
Well, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly generating the output.
Therefore if Groq has achieved significant efficiency improvements, that its, they are not getting that crazy speed by enormous power consumption then maybe they can eventually build low power mass produced cutting edge fabbed chips that run at acceptable speed?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#64This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.
What's the difference in your own words/opinion in quality between GPT-3.5 and GPT-4? For what usecases?
GPT3.5 is great at spitting out marketing babble, summarizing documents and performing superficial analysis but it doesn't take style prompts as well as gpt-4 and its reasoning is significantly worse when you want it to chain of thought follow a complex process while referencing context guidance.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#65In how far is the API compatible with OpenAI? Does it offer logprobs[0] and top_logprobs[1]?
0. https://platform.openai.com/docs/api-reference/chat/create#c...
1. https://platform.openai.com/docs/api-reference/chat/create#c...
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#66Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Friendly fyi - I think this might just be a web interface bug but but I submitted a prompt with the Mixtral model and got a response (great!) then switched the dropdown to Llama and submitted the same prompt and got the exact same response.
It may be caching or it didn't change the model being queried or something else.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#67Not clear if it is due to Groq or to Mixtral, but confident hallucinations are there.
I asked it to come up with name ideas for a company and it hallucinated them successfully :) I think the trick is to know what prompts will likely to yield results that are not likely to be hallucinated. In other contexts it's a feature.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#68Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mobile devices. Or maybe have a dedicated storage, RAM and processing cores that goes as an USB-C add-on? A case with integrated battery and this chip? I'm dreaming of having LLMs on anything. Unlike the "bluetooth on everything" craze, this can be practical as…
Yeah just offload the compute onto the cloud.
Its too unreliable, too restricted and too not-private.
ChatGPT stopped processing images for me, trying to get help but support doesn't appear to be very fast, they asked for more info but not heard back since.
Its too restricted, can't do anything on hard topics. It doesn't work when you try to work out exploits or dangers in a system for example.
Its not private, they say they don't train on API requests but companies steer clear when it comes to send sensitive data.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#69Not clear if it is due to Groq or to Mixtral, but confident hallucinations are there.
I asked it to come up with name ideas for a company and it hallucinated them successfully :) I think the trick is to know what prompts will likely to yield results that are not likely to be hallucinated. In other contexts it's a feature.
A bit of a softball don't you think? The initial message suggests "Are you ready to experience the world's fastest Large Language Model (LLM)? We'd suggest asking about a piece of history"
So I did.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#70This is incredibly fast, indeed. What are the current speeds in T/s for say ChatGPT 3.5 or ChatGPT 4? Just how much faster is this?
I ran the same (Code generation) query and here are my results as end user:
ChaGPT: 1 minute 45 seconds.
Gemini: 16 seconds.
Groq: 3 seconds.