Groq runs Mixtral 8x7B-32k with 500 T/s
1–10 of 482 posts
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#2Incredible tool. The Mixtral 8x7B model running on their hardware did 491.40 T/s for me…
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#3Wow! So fast!
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#4Relevant thread from 5 months ago: https://news.ycombinator.com/item?id=37469434
I'm achieving consistent 450+ tokens/sec for Mixtral 8x7b 32k and ~200 tps for Llama 2 70B-4k.
As an aside, seeing that this is built with flutter Web, perhaps a mobile app is coming soon?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#5Hi folks, I work for Groq. Feel free to ask me any questions.
(If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#6Relevant thread from 5 months ago: https://news.ycombinator.com/item?id=37469434 I'm achieving consistent 450+ tokens/sec for Mixtral 8x7b 32k and ~200 tps for Llama 2 70B-4k. As an aside, seeing that this is built with flutter Web, perhaps a mobile app is coming soon?
There was also another discussion about Groq a couple of months ago https://news.ycombinator.com/item?id=38739199
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#7Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Thank you for doing this AMA
1. How many GroqCards are you using to run the Demo?
2. Is there a newer version you're using which has more SRAM (since the one I see online only has 230MB)? Since this seems to be the number that will drive down your cost (to take advantage of batch processing, CMIIW!)
3. Can TTS pipelines be integrated with your stack? If so, we can truly have very low latency calls!
*Assuming you're using this: https://www.bittware.com/products/groq/
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#8Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)
Thank you for doing this AMA 1. How many GroqCards are you using to run the Demo? 2. Is there a newer version you're using which has more SRAM (since the one I see online only has 230MB)? Since this seems to be the number that will drive down your cost (to take advantage of batch processing, CMIIW!) 3. Can TTS pipelines be integrated with your stack? If so, we can truly have very low latency calls! *Assuming you're u…
1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number.
2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards.
3. Yeah absolutely, see this video of a live demo on CNN!
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#9Earlier quoted context omitted.
Thank you for doing this AMA 1. How many GroqCards are you using to run the Demo? 2. Is there a newer version you're using which has more SRAM (since the one I see online only has 230MB)? Since this seems to be the number that will drive down your cost (to take advantage of batch processing, CMIIW!) 3. Can TTS pipelines be integrated with your stack? If so, we can truly have very low latency calls! *Assuming you're u…
1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number. 2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards. 3. Yeah absolutely,…
Thank you, that demo was insane!
Follow up (noob) question: Are you using a KV cache? That would significantly increase your memory requirements. Or are you forwarding the whole prompt for each auto-regressive pass?
Re: Groq runs Mixtral 8x7B-32k with 500 T/s
#10Earlier quoted context omitted.
1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number. 2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards. 3. Yeah absolutely,…
Thank you, that demo was insane! Follow up (noob) question: Are you using a KV cache? That would significantly increase your memory requirements. Or are you forwarding the whole prompt for each auto-regressive pass?
You're welcome! Yes, we have KV cache. Being able to implement this efficiently in terms of hardware requirements and compute time is one of the benefits of our deterministic chip architecture (and deterministic system architecture).