Is it normal that I have asked two networks (llama/mixtral) the same question ("tell me about most popular audio pitch detection algorithms") and they gave almost the same answer? Both answers start with "Sure, here are some of the most popular pitch detection algorithms used in audio signal processing" and end with "Each of these algorithms has its own strengths and weaknesses, and the choice of algorithm depends on the specific application and the characteristics of the input signal.". And the content is 95% the same. How can it be?
Groq runs Mixtral 8x7B-32k with 500 T/s
281–290 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#282Is it normal that I have asked two networks (llama/mixtral) the same question ("tell me about most popular audio pitch detection algorithms") and they gave almost the same answer? Both answers start with "Sure, here are some of the most popular pitch detection algorithms used in audio signal processing" and end with "Each of these algorithms has its own strengths and weaknesses, and the choice of algorithm depends on…
Yeah it's a bit confusing. See here for details: https://news.ycombinator.com/item?id=39431921
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#283Earlier quoted context omitted.
This is a 50B model. (Mixtral 8x7b)
Oh, sorry, I assumed the 8 was for quantization. 8x7b is a new syntax for me. Still, the NVIDIA chart shows Llama v2 70B at 750 tok/s, no?
I guess that's total throughput, rather than per user? You can increase total throughput by scaling horizontally. You can't increase throughput per user that way.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#284Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
It seems like you are making general purpose chips to run many models. Are we at a stage where we can consider taping out inference networks directly propagating the weights as constants in the RTL design?
Are chips and models obsoleted on roughly the same timelines?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#285The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantee…
Surely once you're scaling over multiple chips/servers/racks you're dealing with retries and checksums and sequence numbers anyway? How do you get around the non-determinacy of networking beyond just hoping that you don't see any errors?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#286Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.
As tome mentioned we don’t quantize, all activations are FP16 And here are some independent benchmarks https://artificialanalysis.ai/models/llama-2-chat-70b
Jesus Christ, these speeds with FP16? That is simply insane.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#287Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#288Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
It seems like you are making general purpose chips to run many models. Are we at a stage where we can consider taping out inference networks directly propagating the weights as constants in the RTL design? Are chips and models obsoleted on roughly the same timelines?
I think the models change far too quickly for that to be viable. A chip has to last several years. Currently we're seeing groundbreaking models released every few months.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#289For me, it was fast when it started printing (it did almost instantly), but it took forever for it to start.
There are a lot of people interested in Groq now, so most jobs are sitting in a queue for a little while.
fair enough... I guess at least it didnt crash like many other overwhelmed sites do... but at the end of the day, it was my experience.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#290The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
Groq devices are really well set up for small-batch-size inference because of the use of SRAM. I'm not so convinced they have a Tok/sec/$ advantage at all, though, and especially at medium to large batch sizes which would be the groups who can afford to buy so much silicon. I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1, and Nvidia cards do get meaningfully higher through…
I've been thinking the same but on the other hand, that would mean they are operating at a huge loss which doesn't scale