I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up by Microsoft, Apple, or Google already for a huge premium.
Groq runs Mixtral 8x7B-32k with 500 T/s
121–130 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#122The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
I honestly don't see the problem.
"just to serve a single model" could be easily fixed by adding a single LPDDR4 channel per LPU. Then you can reload the model sixty times per second and serve 60 different models per second.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#123The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
>The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM [...] IDGAF about any of that, lol. I just want an API endpoint. 480 tokens/sec at $0.27 per million tokens? Sign me in, I don't care about their hardware, at all.
there are providers out there offering for $0 per million tokens, that doesn't mean it is sustainable and won't disappear as soon as the VC well runs dry. Am not saying this is the case for Groq, but in general you probably should care if you want to build something serious on top of anything.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#124So this has nothing to do with `Grok`, the model provided by x.ai? EDIT: Tried using it, very impressed with the speed.
Yeah, it's nothing to do with Elon and we (Groq) had the name first. It's a natural choice of name for something in the field of AI because of the connections to the hacker ethos, but we have the trademark and Elon doesn't. https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/
I mean it sucks that Elon went and claimed Grok when you want Groq, plus you were there first, but getting stuck on the name seems like it's going to be a distraction, so why not choose something different?
When Grok eventually makes the news for some negative thing, so you really want that erroneously associated with your product? Do you really want to pick a fight with the billionaire that owns Twitter, is that a core competency of the company?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#125GPUs are notoriously bad on exploiting sparsity. I wonder if this architecture can do a better job. The groq engineers in this thread, if a neural network had say 60% of its weights set to 0, what would it do to cost & speed in your hardware?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#126Very impressive! I am even more impressed by the API pricing though - 0.27/1M tokens seems like an order of magnitude cheaper than the GPT-3.5 API, and two orders of magnitude cheaper than GPT-4? Am I missing something here?
They’re competing with the lowest cost competitors for mistral atm, which afaik is currently deepinfra at the same pricing
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#127The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
I honestly don't see the problem. "just to serve a single model" could be easily fixed by adding a single LPDDR4 channel per LPU. Then you can reload the model sixty times per second and serve 60 different models per second.
per-chip compute is not the main thing this chip innovates for fast inference, it is the extremely fast memory bandwith. when you do that, you'll loose all of that and will be much worse off than any off the shelf accelerators.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#128Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
As it works at inference do you think 'Representation Engineering ' could be applied to give a sort of fine-tuning ability? https://news.ycombinator.com/item?id=39414532
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#129Earlier quoted context omitted.
> Remember how some British researchers made a self driving car using an LLM? No? Do you mean actual, full self driving on normal roads in traffic?
Yes, IIRC they reason on the car actions using LLMs. They still use image processing but once you identify the objects in the scene, the LLM interprets and decides what to do with the car. I'm not sure which one was it though(Ghost Autonomy maybe?).
Do you have a source? Because that actually, properly working would be headline global news and would value the company in the billions.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#130Do they make money from LLM service or by selling hardware? Homepage is confusing without any reference to other products.