Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

171–180 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#171

Post by gautamcgoel »

tome wrote: post #8
ppsreejith wrote: post #7

Earlier quoted context omitted.

Thank you for doing this AMA 1. How many GroqCards are you using to run the Demo? 2. Is there a newer version you're using which has more SRAM (since the one I see online only has 230MB)? Since this seems to be the number that will drive down your cost (to take advantage of batch processing, CMIIW!) 3. Can TTS pipelines be integrated with your stack? If so, we can truly have very low latency calls! *Assuming you're u…

1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number. 2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards. 3. Yeah absolutely,…

Can you talk about the interconnect? Is it fully custom as well? How do you achieve low latency?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#172

Post by pama »

tome wrote: post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

FYI, I only see a repeating animation and nothing else in my iPhone on lockdown mode, with Safari or Firefox.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#174

Post by brcmthrowaway »

eigenvalue wrote: post #121

I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…

I have it on good authority Apple was very closing to acquiring Groq

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#175

Post by karthityrion »

tome wrote: post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

What is the underlying architecture of the ASICs. Does it use systolic arrays?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#176

Post by neilv »

If the page can't access certain fonts, it will fail to work, while it keeps retrying requests:

    https://fonts.gstatic.com/s/notosansarabic/[...]
    https://fonts.gstatic.com/s/notosanshebrew/[...]
    https://fonts.gstatic.com/s/notosanssc/[...]
(I noticed this because my browser blocks these de facto trackers by default.)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#177

Post by gautamcgoel »

7734128 wrote: post #47
xnorswap wrote: post #38

Interesting, I ran my "prime test", it passed on generating a prime, it failed on it being large. > "Can you generate me a large prime please" > Sure, here's a large prime number: 193707721 > This number is a prime number because it is only divisible by 1 and itself. You can check its primality by trying to divide it by all the numbers up to its square root, and if none of them divide it evenly, then it is a prime nu…

How do you know that it didn't somehow find the largest prime? Perhaps you just threw away a Noble Prize.

Nobel Prize in what? There is no Nobel in mathematics or computer science.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#178

Post by MuffinFlavored »

Cheer2171 wrote: post #132
sorokod wrote: post #93

Earlier quoted context omitted.

Well ok, Groq provides lower latency cheaper access to the same models of questionable quality. Is this not putting lipstick on a pig scenario? I suppose more of a question to pig buyers.

Okay. How about this: Someone posts to HN about an amazing new battery technology, which they demo by showing an average-sized smartphone watching TikTok endlessly scroll for over 500 hours on a single charge. Then someone comments that TikTok is a garbage fire and a horrible corrupting influence, yadda yadda, all that stuff. They ask: what is the point of making phones last longer just to watch TikTok? They say this…

Batteries are useful. The majority of LLMs are not?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#179

Post by karthityrion »

tome wrote: post #167
karthityrion wrote: post #146

Earlier quoted context omitted.

Hi. Are these ASICs only for LLMs or could they accelerate other kinds of models(vision) as well?

It's a general purpose compute engine for numerical computing and linear algebra, so it can accelerate any ML workloads. Previously we've accelerated models for stabilising fusion reactions and for COVID drug discovery * https://alcf.anl.gov/news/researchers-accelerate-fusion-rese... * https://wow.groq.com/groq-accelerates-covid-drug-discovery-3...

So, is this specific chip only for LLMs as the name LPU suggests Language Processing Unit, right?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#180

Post by tome »

gautamcgoel wrote: post #171
tome wrote: post #8

Earlier quoted context omitted.

1. I think our GroqChat demo is using 568 GroqChips. I'm not sure exactly, but it's about that number. 2. We're working on our second generation chip. I don't know how much SRAM it has exactly but we don't need to increase the SRAM to get efficient scaling. Our system is deterministic, which means no need for waiting or queuing anywhere, and we can have very low latency interconnect between cards. 3. Yeah absolutely,…

Can you talk about the interconnect? Is it fully custom as well? How do you achieve low latency?

You can find out about the chip to chip interconnect from our paper below, section 2.3. I don't think that's custom.

We achieve low latency by basically being a software-defined architecture. Our functional units operate completely orthoganal to each other. We don't have to batch in order to achieve parallelism and the system behaviour is completely deterministic, so we can schedule all operations precisely.

https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...

Post reply on HN