Jesus that makes chatgpt and even gemini seem slow AF
Groq runs Mixtral 8x7B-32k with 500 T/s
21–30 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#22Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
@tome for the deterministic system, what if the timing for one chip/part is off due to manufacturing/environmental factors (e.g., temperature) ? How does the system handle this?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#23This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#24Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
You all seem like one of the only companies targeting low-latency inference rather than focusing on throughput (and thus $/inference) - what do you see as your primary market?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#25Earlier quoted context omitted.
They’re for sale on Mouser for $20625 each https://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109... At that price 568 chips would be $11.7M
That seems to be per card instead of chip. I would expect it has multiple chips on a single card.
From the description that doesn't seem to be the case, but I don't know this product well
> Accelerator Cards GroqCard low latency AI/ML Inference PCIe accelerator card with single GroqChip
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#26Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
@tome for the deterministic system, what if the timing for one chip/part is off due to manufacturing/environmental factors (e.g., temperature) ? How does the system handle this?
Those sorts of issues are part of timing analysis for a chip, but once a chip's clock rate is set, they don't really factor in unless there is some kind of dynamic voltage/frequency scaling scheme going on. This chip probably does not do any of that and just uses a fixed frequency, so timing is perfectly predictable.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#27Earlier quoted context omitted.
That seems to be per card instead of chip. I would expect it has multiple chips on a single card.
From the description that doesn't seem to be the case, but I don't know this product well > Accelerator Cards GroqCard low latency AI/ML Inference PCIe accelerator card with single GroqChip
Missed that! Thanks for pointing out!
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#28Earlier quoted context omitted.
Yeah, it's nothing to do with Elon and we (Groq) had the name first. It's a natural choice of name for something in the field of AI because of the connections to the hacker ethos, but we have the trademark and Elon doesn't. https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/
Can't Chamath (he's one of your investors, right), do a thing there? Every person I pitch Groq to is confused and thinks its about Elons unspectacular LLM.
Yeah the confusion has happened a lot to me too. All I know is that it's in the hands of our legal team.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#29Incredible job. Feels dumb or obvious to say this, but this really changes the way I think of using it. The slow autoregression really sucks because it inhibits your ability to skim sections. For me, that creates an unnatural reading environment. This makes chatgpt feel antiqued.
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#30Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
@tome for the deterministic system, what if the timing for one chip/part is off due to manufacturing/environmental factors (e.g., temperature) ? How does the system handle this?
We know the maximum possible clock drift and so we know when we need to do a resynchronisation to keep all the chips in sync. You can read about it in section 3.3 of our recent whitepaper: https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...