Earlier quoted context omitted.
Is that available via an API now?
Kind of, it's in a "Private Preview" with a waitlist.
Via GCP only?
Groq runs Mixtral 8x7B-32k with 500 T/s
141–150 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#142Earlier quoted context omitted.
Friendly fyi - I think this might just be a web interface bug but but I submitted a prompt with the Mixtral model and got a response (great!) then switched the dropdown to Llama and submitted the same prompt and got the exact same response. It may be caching or it didn't change the model being queried or something else.
Thanks, I think it's because the chat context is fed back to the model for the next generation even when you switch models. If you refresh the page that should erase the history and you should get results purely from the model you choose.
Appreciate the quick reply! That's interesting.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#143Earlier quoted context omitted.
Just curious, how does this work out in terms of TCO (even assuming the price of a Groq LPU is 0$)? What you say makes sense, but I'm wondering how you strike a balance between massive horizontal scaling vs vertical scaling. Sometimes (quite often in my experience) having a few beefy servers is much simpler/cheaper/faster than scaling horizontally across many small nodes. Or I got this completely wrong, and your solu…
We're providing by far the lowest latency LLM engine on the planet. You can't reduce latency by scaling horizontally.
Distributed, shared memory machines used to do exactly that in HPC space. They were a NUMA alternative. It works if the processing plus high-speed interconnect are collectively faster than the request rate. The 8x setups with NVLink are kind of like that model.
You may have meant that nobody has a stack that uses clustering or DSM with low-latency interconnects. If so, then that might be worth developing given prior results in other low-latency domains.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#144I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…
Really glad you like it! We've been working hard on it.
The speed part or the being swallowed part?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#145Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.
At least for the earlier Llama 70B demo, they claimed to be running unquantized. https://twitter.com/lifebypixels/status/1757619926360096852 Update: This comment says "some data is stored as FP8 at rest" and I don't know what that means. https://news.ycombinator.com/item?id=39432025
The weights are quantized to FP8 when they're stored in memory, but all the activations are computed at full FP16 precision.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#146Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Hi. Are these ASICs only for LLMs or could they accelerate other kinds of models(vision) as well?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#147This demo has more than 500 chips btw, it’s not exactly an apples to apples comparison with 1 GPU…
Definitely not, but even with a comparison to 500 GPUs Groq will still come out on top because you can never reduce latency by adding more parallel compute :)
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#148Earlier quoted context omitted.
We run the open source models that everyone else has access to. What we're trying to show off is our low latency and high throughput, not the model itself.
But if the model is useless/full of hallucinations, why does the speed of its output matter? "generate hallucinated results, faster"
[deleted]
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#149Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
If I understand correctly, you're using specialized hardware to improve token generation speed, which is very latency bound on the speed of computation. However generating tokens only requires multiplying 1-dimensional matrices usually. If I enter a prompt with ~100 tokens then your service goes much slower. Probably because you have to multiply 2-dimensional matrices. What are you doing to improve the computation speed of prompt processing?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#150Earlier quoted context omitted.
We run the open source models that everyone else has access to. What we're trying to show off is our low latency and high throughput, not the model itself.
But if the model is useless/full of hallucinations, why does the speed of its output matter? "generate hallucinated results, faster"
