Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

111–120 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#111

Post by imiric »

Impressive demo!

However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists?

Also, while the CNN Vapi demo was impressive as well, a few weeks ago here[1] someone shared https://smarterchild.chat/. That also has _very_ low audio latency, making natural conversation possible. From that discussion it seems that https://www.sindarin.tech/ is behind it. Do we know if they use Groq LPUs or something else?

I think that once you reach ~50 t/s, real-time interaction is possible. Anything higher than that is useful for generating large volumes of data quickly, but there are diminishing returns as it's far beyond what humans can process. Maybe such speeds would be useful for AI-AI communication, transferring knowledge/context, etc.

So an LPU product that's only focused on AI-human interaction could have much lower capabilities, and thus much lower cost, no?

[1]: https://news.ycombinator.com/item?id=39180237

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#113

Post by trsohmers »

treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq states in this article [0] that they used 576 chips to achieve these results, and continuing with your analysis, you also need to factor in that for each additional user you want to have requires a separate KV cache, which can add multiple more gigabytes per user.

My professional independent observer opinion (not based on my 2 years of working at Groq) would have me assume that their COGS to achieve these performance numbers would exceed several million dollars, so depreciating that over expected usage at the theoretical prices they have posted seems impractical, so from an actual performance per dollar standpoint they don’t seem viable, but do have a very cool demo of an insane level of performance if you throw cost concerns out the window.

[0]: https://www.nextplatform.com/2023/11/27/groq-says-it-can-dep...

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#114

Post by tome »

kkzz99 wrote: post #107
tome wrote: post #98

Earlier quoted context omitted.

We build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits of low latency.

How does that work when the card only has 230MB of SRAM?

We connect hundreds of chips across several racks with fast interconnect.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#115

Post by mrtksn »

tome wrote: post #101
mrtksn wrote: post #97

Earlier quoted context omitted.

100s of chips for who knows how many clients. The mobile phone will have to do calculations just for 1 client.

Yes, we pipeline requests so multiple users are being handled by the same hardware at one time.

Thanks for the clarification. So, would you say that Groq has a potential to have let's say OpenAI speeds on handheld devices at reasonable energy consumption? Or is that not really what this tech's strength is maybe?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#116

Post by tome »

imiric wrote: post #111

Impressive demo! However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? Also, while the CNN Vapi demo was impressive as well, a few weeks ago here[1] someone shared https://smarterchild.chat/ . That also has _very_ low audio latency, making natural conversation possible. From that discussion it seems tha…

> However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists?

For API access to our tokens as a service we guarantee to beat any other provider on cost per token (see https://wow.groq.com). In terms of selling hardware, we're focused on selling whole systems, and they're only really suitable for corporations or research institutions.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#117

Post by tome »

mrtksn wrote: post #115
tome wrote: post #101

Earlier quoted context omitted.

Yes, we pipeline requests so multiple users are being handled by the same hardware at one time.

Thanks for the clarification. So, would you say that Groq has a potential to have let's say OpenAI speeds on handheld devices at reasonable energy consumption? Or is that not really what this tech's strength is maybe?

The industry as a whole is a very long way away from that. The power requirements are too high for mobile.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#118

Post by mrtksn »

SahAssar wrote: post #105
mrtksn wrote: post #41

Does this make it practical to run LLMs on mobile devices? I wonder about the power consumption and if it can make sense to have it integrated in some future mobile devices. Or maybe have a dedicated storage, RAM and processing cores that goes as an USB-C add-on? A case with integrated battery and this chip? I'm dreaming of having LLMs on anything. Unlike the "bluetooth on everything" craze, this can be practical as…

> Remember how some British researchers made a self driving car using an LLM? No? Do you mean actual, full self driving on normal roads in traffic?

Yes, IIRC they reason on the car actions using LLMs. They still use image processing but once you identify the objects in the scene, the LLM interprets and decides what to do with the car.

I'm not sure which one was it though(Ghost Autonomy maybe?).

Post reply on HN