Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

421–430 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#421

Post by doctorpangloss »

monkmartinez wrote: post #390
a_wild_dandan wrote: post #256

Earlier quoted context omitted.

Is your confidence rooted in quantified testing, or just vibes? I'm sure you're right, just curious. (My reasoning: running inference at full fp16 is borderline wasteful. You can use q7 with almost no loss.)

I know some fancy benchmark says "almost no loss", but... subjectively, there is a clear quality loss. You can try for yourself, I can run Mixtral at 5.8bpw and there is an OBVIOUS difference between what I have seen from Groq and my local setup beside the sound barrier shattering speed of Groq. I didn't know Mixtral could output such nice code and I have used it A LOT locally.

Yes, but this gray area underperformance that lets them claim they are the cheapest and fastest appeals to people for whom qualitative (aka real) performance doesn’t matter.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#422

Post by ndjdbdjdbev »

LtdJorge wrote: post #359
frozenport wrote: post #197

Earlier quoted context omitted.

Tip of the ice-berg. DRAM needs to be refreshed every X cycles. This means you don't know the time it takes to read from memory. You could be reading at a refresh cycle. This circuitry also adds latency.

OP says SRAM, which doesn't decay so no refreshing.

Timing can simply mean the FETs that make up the logic circuits of a chip. The transition from high to low and low to high has a minimum safe time to register properly...

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#423

Post by tome »

fennecbutt wrote: post #386
matanyall wrote: post #94

Earlier quoted context omitted.

Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configur…

Are there voice responses in the demo? I couldn't find em?

Here's a live demo of CNN of Groq plugged into a voice API

https://www.youtube.com/watch?v=pRUddK6sxDg&t=235s

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#424

Post by tome »

dsrtslnd23 wrote: post #396
tome wrote: post #116

Earlier quoted context omitted.

> However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? For API access to our tokens as a service we guarantee to beat any other provider on cost per token (see https://wow.groq.com ). In terms of selling hardware, we're focused on selling whole systems, and they're only really suitable for corporations…

How open is your early access? i.e. likelihood to get API access granted right now

We are absolutely slammed with requests right now, so I don't know, sorry.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#425

Post by tome »

pwillia7 wrote: post #358
tome wrote: post #116

Earlier quoted context omitted.

> However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? For API access to our tokens as a service we guarantee to beat any other provider on cost per token (see https://wow.groq.com ). In terms of selling hardware, we're focused on selling whole systems, and they're only really suitable for corporations…

Do you have any data on how many more tokens I would use with the increased speed? In the demo alone I just used way more tokens than I normally would testing an LLM since it was so amazingly fast.

Interesting question! Hopefully being faster is so much more useful to you that you use a lot more :)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#427

Post by tome »

johndough wrote: post #303
tome wrote: post #162

Earlier quoted context omitted.

I don't think it should be quadratic in input length. Why do you think it is?

You can ask your website: "What is the computational complexity of self-attention with respect to input sequence length?" It'll answer something along the lines of self-attention being O(n^2) (where n is the sequence length) because you have to compute an attention matrix of size n^2. There are other attention mechanisms with better computational complexity, but they usually result in worse large language models. To…

OK, thanks, that's useful to know. Personally I'm not involved directly in implementing the model, so I don't know what we do there.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#430

Post by mike_hearn »

bethekind wrote: post #412
foundval wrote: post #324

Earlier quoted context omitted.

(Groq Employee) Agreed, one should care, and especially since this particular service is very differentiated by its speed and has no competitors. That being said, until there's another option at anywhere that speed.. That point is moot, isn't it :) For now, Groq is the only option that can let you build an UX with near-instant response times. Or a live agents that help with a human-to-human interaction. I could go on…

Why go so fast? Aren't Nvidias products fast enough from a TPS perspective?

OpenAI have a voice powered chat mode in their app and there's a noticeable delay of a few seconds between finishing your sentence and the bot starting to speak.

I think the problem is that for realistic TTS you need quite a few tokens because the prosody can be affected by tokens that come a fair bit further down the sentence, consider the difference in pitch between:

"The war will be long and bloody"

vs

"The war will be long and bloody?"

So to begin TTS you need quite a lot of tokens, which in turn means you have to digest the prompt and run a whole bunch of forward passes before you can start rendering. And of course you have to keep up with the speed of regular speech, which OpenAI sometimes struggles with.

That said, the gap isn't huge. Many apps won't need it. Some use cases where low latency might matter:

- Phone support.

- Trading. Think digesting a press release into an action a few seconds faster than your competitors.

- Agents that listen in to conversations and "butt in" when they have something useful to say.

- RPGs where you can talk to NPCs in realtime.

- Real-time analysis of whatever's on screen on your computing device.

- Auto-completion.

- Using AI as a general command prompt. Think AI bash.

Undoubtably there will be a lot more though. When you give people performance, they find ways to use it.

Post reply on HN