Please when/where can I buy some of these for home use? Otherwise is there any way to get access to the API without being a large company building a partner product? I would love this for personal use.
Groq runs Mixtral 8x7B-32k with 500 T/s
311–320 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#312Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#313The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
Groq devices are really well set up for small-batch-size inference because of the use of SRAM. I'm not so convinced they have a Tok/sec/$ advantage at all, though, and especially at medium to large batch sizes which would be the groups who can afford to buy so much silicon. I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1, and Nvidia cards do get meaningfully higher through…
(Groq Employee) It's hard to discuss Tok/sec/$ outside of the context of a hardware sales engagement.
This is because the relationship between Tok/s/u, Tok/s/system, Batching, and Pipelining is a complex one that involves compute utilization, network utilization, and (in particular) a host of compilation techniques that we wouldn't want to share publicly. Maybe we'll get to that level of transparency at some point, though!
As far as Batching goes, you should consider that with synchronous systems, if all the stars align, Batch=1 is all you need. Of course, the devil is in the details, and sometimes small batch numbers still give you benefits. But Batch 100's generally gives no advantages. In fact, the entire point of developing deterministic hardware and synchronous systems is to avoid batching in the first place.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#314The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
Groq devices are really well set up for small-batch-size inference because of the use of SRAM. I'm not so convinced they have a Tok/sec/$ advantage at all, though, and especially at medium to large batch sizes which would be the groups who can afford to buy so much silicon. I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1, and Nvidia cards do get meaningfully higher through…
I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#315Holy smokes this is fast
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#316So this has nothing to do with `Grok`, the model provided by x.ai? EDIT: Tried using it, very impressed with the speed.
Thanks for asking just the question I wanted to ask!
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#317Sorry if this is dumb but how is this different to Elons Grok? Was Groq chosen as a joke or homage ?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#318Sorry if this is dumb but how is this different to Elons Grok? Was Groq chosen as a joke or homage ?
This company is older than Elon's
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#319Earlier quoted context omitted.
LLM inference is inherently a sequential problem. You can't speed it up by doing more in parallel. You can't generate the 101st token before you've generated the 100th.
Technically, I guess you can use speculative execution to speed it up, and in that way take a guess at what the 100th token will be and start on the 101st token at the same time? Though it probably has it's own unforeseen challenges. Everything is predictable with enough guesses.
People are pretty cagey about what they use in production, but yes, speculative sampling can offer massive speedups in inference
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#320Earlier quoted context omitted.
Yup, understood. Access to consumer hardware like this is not something that we provide at the moment, I'm afraid.
Don't blame you. Been at plenty of startups, resources are finite, and focus is important. My only point was to, well, perhaps bump this up from #100 on your personal priority list perhaps to #87, to the limited extent that influences your business.
Groq Engineer here as well; we actually built our compiler to compile pytorch, TensorFlow, and Onnx natively, so a lot of the amazing work being done by y'all isn't building much of a moat. We got LLama2 working on our hardware in just a couple of days!