Switching the model between Mixtral and Llama I get word for word the same responses. Is this expected?
No…
Groq runs Mixtral 8x7B-32k with 500 T/s
81–90 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#82Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
are your accelerator chips designed in-house? or they're some specialized silicon or FPGPU or something that you wrote very optimized code for inference? it's really amazing! the first time I tried the demo, I had to try a few prompts to believe it wasn't just an animation :)
Yup, custom ASIC, designed in-house, built into a system of several racks, hundreds of chips, with fast interconnect. Really glad you enjoyed it!
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#83Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
When will we be able to buy Groq accelerator cards that would be affordable for hobbyists?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#84Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#85Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
You're running fp32 models, fp16 or quantized?
FP16 for calculating all activations. Some data is stored as FP8 at rest.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#86The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…
Groq devices are really well set up for small-batch-size inference because of the use of SRAM.
I'm not so convinced they have a Tok/sec/$ advantage at all, though, and especially at medium to large batch sizes which would be the groups who can afford to buy so much silicon.
I assume given the architecture that Groq actually doesn't get any faster for batch sizes >1, and Nvidia cards do get meaningfully higher throughput as batch size gets into the 100's.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#87Earlier quoted context omitted.
I assume this is a million-dollar rack of custom chips so it's probably not coming to mobile any time soon.
Well, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly generating the output. Therefore if Groq has achieved significant efficiency improvements, that its, they are not getting that crazy speed by enormous power consumption then maybe they can eventually build low power mass produced cutting edge fabbed chips that run…
The thing is, I don't see any efficiency improvements. I see models running fast on very expensive hardware using techniques that don't scale down.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#88Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
How does the Groq PCIE Card work exactly? Does it use system ram to stream the model data to the card? How many T/s could one expect with e.g. 36000Mhz DDR4 Ram?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#89Earlier quoted context omitted.
Well, currently its entirely possible to run these models on iPhones. It's just not practical because it eats all the resources and the battery when slowly generating the output. Therefore if Groq has achieved significant efficiency improvements, that its, they are not getting that crazy speed by enormous power consumption then maybe they can eventually build low power mass produced cutting edge fabbed chips that run…
The thing is, I don't see any efficiency improvements. I see models running fast on very expensive hardware using techniques that don't scale down.
Care to explain? Are they using 10x energy for 10x speed improvements?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#90In how far is the API compatible with OpenAI? Does it offer logprobs[0] and top_logprobs[1]? 0. https://platform.openai.com/docs/api-reference/chat/create#c... 1. https://platform.openai.com/docs/api-reference/chat/create#c...
You can find our API docs here, including details of our OpenAI compatibility