Earlier quoted context omitted.
The speed part or the being swallowed part?
The speed part. We're not interested in being swallowed. The aim is to be bigger than Nvidia in three years :)
Go for it!
Groq runs Mixtral 8x7B-32k with 500 T/s
161–170 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#162Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
If I understand correctly, you're using specialized hardware to improve token generation speed, which is very latency bound on the speed of computation. However generating tokens only requires multiplying 1-dimensional matrices usually. If I enter a prompt with ~100 tokens then your service goes much slower. Probably because you have to multiply 2-dimensional matrices. What are you doing to improve the computation sp…
I don't think it should be quadratic in input length. Why do you think it is?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#163Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
How well would your hardware work for image/video generation?
It should work great as far as I know. We've implemented some diffusion models for image generation but we don't offer them at the moment. I'm not aware of us having implemented any video models.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#164The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
> more than a single model and a lot of finetunes/high rank LoRAs
I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using barely more ram than the base model alone.
The fine-tuning could perhaps be done in such a way that only perhaps 0.1% of the weights are changed, and for every computation the difference is computed not over the weights, but of the output layer activations.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#165Earlier quoted context omitted.
Thanks, I think it's because the chat context is fed back to the model for the next generation even when you switch models. If you refresh the page that should erase the history and you should get results purely from the model you choose.
Appreciate the quick reply! That's interesting.
You're welcome. Thanks for reporting. It's pretty confusing so maybe we should change it :)
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#166I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…
Sure, but the responses are very poor compared to MS tools.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#167Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Hi. Are these ASICs only for LLMs or could they accelerate other kinds of models(vision) as well?
It's a general purpose compute engine for numerical computing and linear algebra, so it can accelerate any ML workloads. Previously we've accelerated models for stabilising fusion reactions and for COVID drug discovery
* https://alcf.anl.gov/news/researchers-accelerate-fusion-rese...
* https://wow.groq.com/groq-accelerates-covid-drug-discovery-3...
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#168Earlier quoted context omitted.
You can find our API docs here, including details of our OpenAI compatibility https://docs.api.groq.com/
Filled the form for API Access last night. Is there a delay with increased demand now?
Yes, there's a huge amount of demand because Twitter discovered us yesterday. There will be a backlog, so sorry about that.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#169Really impressive work! I wonder how easy would it be to support (a future open source version of) SORA using Groq's design. Will there be a Video Processing Unit (VPU)?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#170any noscript/basic (x)html prompt?