Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

11–20 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#11

Post by ppsreejith »

tome wrote: post #10
ppsreejith wrote: post #9

Earlier quoted context omitted.

Thank you, that demo was insane! Follow up (noob) question: Are you using a KV cache? That would significantly increase your memory requirements. Or are you forwarding the whole prompt for each auto-regressive pass?

You're welcome! Yes, we have KV cache. Being able to implement this efficiently in terms of hardware requirements and compute time is one of the benefits of our deterministic chip architecture (and deterministic system architecture).

Thanks again! Hope I'm not overwhelming but one more question: Are you decoding with batch size = 1 or is it more?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#12

Post by tome »

ppsreejith wrote: post #11
tome wrote: post #10

Earlier quoted context omitted.

You're welcome! Yes, we have KV cache. Being able to implement this efficiently in terms of hardware requirements and compute time is one of the benefits of our deterministic chip architecture (and deterministic system architecture).

Thanks again! Hope I'm not overwhelming but one more question: Are you decoding with batch size = 1 or is it more?

That's OK, feel free to keep asking!

I think currently 1. Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to deliver good throughput even at batch size 1.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#13

Post by WiSaGaN »

tome wrote: post #8
ppsreejith wrote: post #7

Earlier quoted context omitted.

Thank you for doing this AMA 1. How many GroqCards are you using to run the Demo? 2. Is there a newer version you're using which has more SRAM (since the one I see online only has 230MB)? Since this seems to be the number that will drive down your cost (to take advantage of batch processing, CMIIW!) 3. Can TTS pipelines be integrated with your stack? If so, we can truly have very low latency calls! *Assuming you're u…

1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number. 2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards. 3. Yeah absolutely,…

How much do 568 chips cost? What’s the cost ratio of it comparing to setup with roughly the same throughput using A100?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#15

Post by Oras »

tome wrote: post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

Impressive speed. Are there any plans to run fine-tuned models?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#16

Post by benchess »

WiSaGaN wrote: post #13
tome wrote: post #8

Earlier quoted context omitted.

1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number. 2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards. 3. Yeah absolutely,…

How much do 568 chips cost? What’s the cost ratio of it comparing to setup with roughly the same throughput using A100?

They’re for sale on Mouser for $20625 each https://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109...

At that price 568 chips would be $11.7M

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#17

Post by tome »

sebzim4500 wrote: post #14

So this has nothing to do with `Grok`, the model provided by x.ai? EDIT: Tried using it, very impressed with the speed.

Yeah, it's nothing to do with Elon and we (Groq) had the name first. It's a natural choice of name for something in the field of AI because of the connections to the hacker ethos, but we have the trademark and Elon doesn't.

https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#18

Post by tome »

benchess wrote: post #16
WiSaGaN wrote: post #13

Earlier quoted context omitted.

How much do 568 chips cost? What’s the cost ratio of it comparing to setup with roughly the same throughput using A100?

They’re for sale on Mouser for $20625 each https://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109... At that price 568 chips would be $11.7M

Yeah, I don't know what the cost to us is to build out our own hardware but it's significantly less expensive than retail.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#19

Post by WiSaGaN »

benchess wrote: post #16
WiSaGaN wrote: post #13

Earlier quoted context omitted.

How much do 568 chips cost? What’s the cost ratio of it comparing to setup with roughly the same throughput using A100?

They’re for sale on Mouser for $20625 each https://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109... At that price 568 chips would be $11.7M

That seems to be per card instead of chip. I would expect it has multiple chips on a single card.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#20

Post by terhechte »

tome wrote: post #17
sebzim4500 wrote: post #14

So this has nothing to do with `Grok`, the model provided by x.ai? EDIT: Tried using it, very impressed with the speed.

Yeah, it's nothing to do with Elon and we (Groq) had the name first. It's a natural choice of name for something in the field of AI because of the connections to the hacker ethos, but we have the trademark and Elon doesn't. https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/

Can't Chamath (he's one of your investors, right), do a thing there? Every person I pitch Groq to is confused and thinks its about Elons unspectacular LLM.
Post reply on HN