Not clear if it is due to Groq or to Mixtral, but confident hallucinations are there.
At top left hand corner you can change the model to Llama2 70B Model.
Groq runs Mixtral 8x7B-32k with 500 T/s
71–80 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#72Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.
As a fellow scientist I concur with the approach of skepticism by default. Our chat app and API are available for everyone to experiment with and compare output quality with any other provider.
I hope you are enjoying your time of having an empty calendar :)
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#73This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.
I wonder if Gemini Pro 1.5 will act as a forcing function to lower GPT4 pricing.
Is that available via an API now?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#74Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Friendly fyi - I think this might just be a web interface bug but but I submitted a prompt with the Mixtral model and got a response (great!) then switched the dropdown to Llama and submitted the same prompt and got the exact same response. It may be caching or it didn't change the model being queried or something else.
Thanks, I think it's because the chat context is fed back to the model for the next generation even when you switch models. If you refresh the page that should erase the history and you should get results purely from the model you choose.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#75Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
You all seem like one of the only companies targeting low-latency inference rather than focusing on throughput (and thus $/inference) - what do you see as your primary market?
Yes, because we're one of the only companies whose hardware can actually support low latency! Everyone else is stuck with traditional designs and they try to make up for their high latency by batching to get higher throughput. But not all applications work with high throughput/high latency ... Low latency unlocks feeding the result of one model into the input of another model. Check out this conversational AI demo on CNN. You can't do that kind of thing unless you have low latency.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#76Switching the model between Mixtral and Llama I get word for word the same responses. Is this expected?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#77Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Is it possible to buy Groq chips and how much do they cost?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#78Switching the model between Mixtral and Llama I get word for word the same responses. Is this expected?
Yeah, this is a common observation. See my comment at https://news.ycombinator.com/item?id=39431921
Maybe we should change the behavior to stop people getting confused.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#79Earlier quoted context omitted.
We run the open source models that everyone else has access to. What we're trying to show off is our low latency and high throughput, not the model itself.
But if the model is useless/full of hallucinations, why does the speed of its output matter? "generate hallucinated results, faster"
No, it is "do whatever you were already doing with ML, faster"
This question seems either from a place of deep confusion or is in bad faith. This post is about hardware. The hardware is model independent.* Any issues with models, like hallucinations, are going to be identical if it is run on this platform or a bunch of Nvidia GPUs. Performance in terms of hardware speed and efficiency are orthogonal to performance in terms of model accuracy and hallucinations. Progress on one axis can be made independently to the other.
* Technically no, but close enough
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#80The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server rack density) and serve these models reasonably well.
It might work well if you have a single model with lots of customers, but as soon as you need more than a single model and a lot of finetunes/high rank LoRAs etc., these won't be usable. Or for any on-prem deployment since the main advantage is consolidating people to use the same model, together.
[0]: https://wow.groq.com/groqcard-accelerator/
[1]: https://twitter.com/tomjaguarpaw/status/1759615563586744334