Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

261–270 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#261

Post by nickpsecurity »

tome wrote: post #158
nickpsecurity wrote: post #143

Earlier quoted context omitted.

Distributed, shared memory machines used to do exactly that in HPC space. They were a NUMA alternative. It works if the processing plus high-speed interconnect are collectively faster than the request rate. The 8x setups with NVLink are kind of like that model. You may have meant that nobody has a stack that uses clustering or DSM with low-latency interconnects. If so, then that might be worth developing given prior…

I think existing players will have trouble developing a low latency solution like us whilst they are still running on non-deterministic hardware.

While you’re here, I have a quick, off-topic question. We‘ve seen incredible results with GPT3-176B (Davinci) and GPT4 (MoE). Making attempts at open models that reuse their architectural strategies could have a high impact on everyone. Those models took 2500-25000 GPU’s to train, though. It would be great to have a low-cost option for pre training Davinci-class models.

It would great if a company or others with AI hardware were willing to do production runs of chips sold at cost specifically to make open, permissive-licensed models. As in, since you’d lose profit, the cluster owner and users would be legally required to only make permissive models. Maybe at least one in each category (eg text, visual).

Do you think your company or any other hardware supplier would do that? Or someone sell 2500 GPU’s at cost for open models?

(Note to anyone involved in CHIPS Act: please fund a cluster or accelerator specifically for this.)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#264

Post by snowfield »

SeanAnderson wrote: post #236
MasterScrat wrote: post #226

Earlier quoted context omitted.

Scaling up compute can improve throughput, but can't easily improve latency between tokens. Generation is usually bottlenecked by the time it takes to go through the network for each token. To speed that up, you need to perform these computations faster, which is a hard problem after you've exhausted all the obvious options (use the fastest accelerator you can find, cache what you can etc).

Yeah. That makes sense, thank you for clarifying. I updated my original post with a chart from NVIDIA which highlights the H100's capabilities. It doesn't seem unreasonable to expect a 7B model to run at 500 tok/s on that hardware.

This is a 50B model. (Mixtral 8x7b)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#265

Post by throwawayurlife »

frognumber wrote: post #204
KRAKRISMOTT wrote: post #200

Earlier quoted context omitted.

You have to prove the OP had personal gains. If he's just a troll, it will be difficult.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information.

I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#266

Post by stormfather »

imiric wrote: post #111

Impressive demo! However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? Also, while the CNN Vapi demo was impressive as well, a few weeks ago here[1] someone shared https://smarterchild.chat/ . That also has _very_ low audio latency, making natural conversation possible. From that discussion it seems tha…

>>50 t/s is absolutely necessary for real-time interaction with AI systems. Most of the LLM's output will be internal monologue and planning, performing RAG and summarization, etc, with only the final output being communicated to you. Imagine a blazingly fast GPT-5 that goes through multiple cycles of planning out how to answer you, searching the web, writing book reports, debating itself, distilling what it finds, critiquing and rewriting its answer, all while you blink a few times.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#267

Post by mirekrusin »

tome wrote: post #201
vimarsh6739 wrote: post #196

Earlier quoted context omitted.

Might be a bit out of context, but isn't the TPU also optimized for low latency inference? (Judging by reading the original TPU architecture paper here - https://arxiv.org/abs/1704.04760 ). If so, does Groq actually provide hardware support for LLM inference?

Jonathan Ross on that paper is Groq's founder and CEO. Groq's LPU is an natural continuation of the breakthrough ideas he had when designing Google's TPU. Could you clarify your question about hardware support? Currently we build out our hardware to support our cloud offering, and we sell systems to enterprise customers.

Are you considering targeting consumer market? There are a lot of people throwing $2k-$4k into local setups and they primarily care about inference.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#268

Post by tome »

nickpsecurity wrote: post #261
tome wrote: post #158

Earlier quoted context omitted.

I think existing players will have trouble developing a low latency solution like us whilst they are still running on non-deterministic hardware.

While you’re here, I have a quick, off-topic question. We‘ve seen incredible results with GPT3-176B (Davinci) and GPT4 (MoE). Making attempts at open models that reuse their architectural strategies could have a high impact on everyone. Those models took 2500-25000 GPU’s to train, though. It would be great to have a low-cost option for pre training Davinci-class models. It would great if a company or others with AI h…

Great idea, but Groq doesn't have a product suitable for training at the moment. Our LPUs shine in inference.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#269

Post by bearjaws »

sp332 wrote: post #106
karpathy wrote: post #59

Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.

At least for the earlier Llama 70B demo, they claimed to be running unquantized. https://twitter.com/lifebypixels/status/1757619926360096852 Update: This comment says "some data is stored as FP8 at rest" and I don't know what that means. https://news.ycombinator.com/item?id=39432025

Nothing really wrong with FP8 IMO, it performs pretty damn well usually within 98% while significantly reducing memory usage.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#270

Post by tome »

mirekrusin wrote: post #267
tome wrote: post #201

Earlier quoted context omitted.

Jonathan Ross on that paper is Groq's founder and CEO. Groq's LPU is an natural continuation of the breakthrough ideas he had when designing Google's TPU. Could you clarify your question about hardware support? Currently we build out our hardware to support our cloud offering, and we sell systems to enterprise customers.

Are you considering targeting consumer market? There are a lot of people throwing $2k-$4k into local setups and they primarily care about inference.

At the moment we're concentrating on building out our API and serving the enterprise market.
Post reply on HN