Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

341–350 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#341

Post by Aeolun »

trsohmers wrote: post #113
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq states in this article [0] that they used 576 chips to achieve these results, and continuing with your analysis, you also need to factor in that for each additional user you want to have requires a separate KV cache, which can add multiple more gigabytes per user. My professional independent observer opinion (not based on my 2 years of working at Groq) would have me assume that their COGS to achieve these perfor…

I think that just means it’s for people that really want it?

John doe and his friends will never have a need to have their fart jokes generated at this speed, and are more interested in low costs.

But we’d recently been doing call center operations and being able to quickly figure out what someone said was a major issue. You kind of don’t want your system to wait for a second before responding each time. I can imagine it making sense if it reduces the latency to 10ms there as well. Though you might still run up against the ‘good enough’ factor.

I guess few people want to spend millions to go from 1000ms to 10ms, but when they do they really want it.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#342

Post by windexh8er »

throwawayurlife wrote: post #265
frognumber wrote: post #204

Earlier quoted context omitted.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."

Feel free to share some legal precedence where this situation has fared poorly for someone who "overheard it at a bar".

It'd also be a good time to watch you lose all that money on your hokey-pokey assumption.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#343

Post by frozenport »

jonplackett wrote: post #340
tome wrote: post #139

Earlier quoted context omitted.

Really glad you like it! We've been working hard on it.

Is this useful for training as well as running a model. Or is this approach specifically for running an already-trained model faster?

In principle, training is basically the same as running inference but iteratively, in practice training would use a different software stack.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#344

Post by sexy_seedbox »

tagyro wrote: post #275

I (only) ran a couple of prompts but I am impressed. It has the speed of gpt 3.5 and the quality of gpt 4. Seriously considering switching from [open]AI to Mix/s/tral in my apps.

Try more prompts, both models could not even answer the "Sally has 3 brothers" question; really disappointing.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#345

Post by tome »

jonplackett wrote: post #340
tome wrote: post #139

Earlier quoted context omitted.

Really glad you like it! We've been working hard on it.

Is this useful for training as well as running a model. Or is this approach specifically for running an already-trained model faster?

Currently graphics processors work well for training. Language processors (LPUs) excel at inference.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#346

Post by tandr »

Klaus23 wrote: post #199

The demo is pretty cool, but the mobile interface could be a parody of bad interface design. The text box at the top is hard to reach if you want to open the keyboard, which automatically closes, or press the button to send the question, and the chat history is out of chronological order for no logical reason. Edit: Text selection is also broken.

> Edit: Text selection is also broken.

Or disabled?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#347

Post by frognumber »

foundval wrote: post #308
frognumber wrote: post #205

Earlier quoted context omitted.

From a theoretical perspective, this is absolutely not true. Asynchronous logic can achieve much lower latency guarantees than synchronous logic. Come to think of it, this is one of the few places where asynchronous logic might be more than academic... Async logic is hard with complex control flows, which deep learning inference does not have. (From a practical perspective, I know you were comparing to independently-…

(Groq Employee) You're right - we are comparing to independently-clocked logic. I wonder whether async logic would be feasible for reconfigurable "Spatial Processor" type architectures [1]. As far as LPU architectures go, they fall in the "Matrix of Processing Engines"[1] family of architectures, which I would naively guess is not the best suited to leverage async logic. 1: I'm using the "Spatial Processor" (7:14) an…

Curiously, almost all of this video is mostly covered by computer architectures lit in the late 90's early 00's. At the time, I recall Tom Knight had done most of the analysis in this video, but I don't know if he ever published it. It was extrapolating into the distant future.

To answer your questions:

- Spatial processors are an insanely good fit for async logic

- Matrix of processing engines are a moderately good fit -- definitely could be done, but I have no clue if it'd be a good idea.

In SP, especially in an ASIC, each computation can start as soon as the previous one finishes. If you have a 4-bit layer, and 8-bit layer, and a 32-bit layer, those will take different amounts of time to run. Individual computations can take different amounts of time too (e.g. an ADD with a lot of carries versus one with just a few). In an SP, a compute will take as much time as it needs, and no more.

Footnote: Personally, I think there are a lot of good ideas in 80's era and earlier processors for the design of individual compute units which have been forgotten. The basic move in architectures up through 2005 was optimizing serial computation speed at the cost of power and die size (Netburst went up to 3.8GHz two decades ago). With much simpler old-school compute units, we can have *many* more of them than a modern multiply unit. Critically, they could be positioned closer to the data, so there would be less data moving around. Especially the early pipelined / scalar / RISC cores seem very relevant. As a point of reference, a 4090 has 16k CUDA cores running at just north of 2GHz. It has the same number of transistors as 32,000 SA-110 processors (running at 200MHz on a 350 nanometer process in 1994).

TL;DR: I'm getting old and either nostalgic or grumpy. Dunno which.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#348

Post by Jensson »

ggnore7452 wrote: post #242

The Groq demo was indeed impressive. I work with LLM alot in work, and a generation speed of 500+ tokens/s would definitely change how we use these products. (Especially considering it's an early-stage product) But the "completely novel silicon architecture" and the "self-developed LPU" (claiming not to use GPUs)... makes me bit skeptical. After all, pure speed might be achievable through stacking computational power…

> Shouldn't innovation at the GPU level be quite challenging, especially to achieve such groundbreaking speeds?

GPUs are general purpose, a for purpose built chip that is better isn't that hard to make at all. Google didn't have to work hard at all to invent TPUs which is that idea as well, they said their first tests proved the idea worked so it didn't require anything near Nvidias scale or expertise.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#349

Post by frozenport »

frognumber wrote: post #347
foundval wrote: post #308

Earlier quoted context omitted.

(Groq Employee) You're right - we are comparing to independently-clocked logic. I wonder whether async logic would be feasible for reconfigurable "Spatial Processor" type architectures [1]. As far as LPU architectures go, they fall in the "Matrix of Processing Engines"[1] family of architectures, which I would naively guess is not the best suited to leverage async logic. 1: I'm using the "Spatial Processor" (7:14) an…

Curiously, almost all of this video is mostly covered by computer architectures lit in the late 90's early 00's. At the time, I recall Tom Knight had done most of the analysis in this video, but I don't know if he ever published it. It was extrapolating into the distant future. To answer your questions: - Spatial processors are an insanely good fit for async logic - Matrix of processing engines are a moderately good…

This was sort of the dream of KNL but today I noticed

    Xeon Phi CPUs support (a.k.a. Knight Landing and Knight Mill) are marked as deprecated. GCC will emit a warning when using the -mavx5124fmaps, -mavx5124vnniw, -mavx512er, -mavx512pf, -mprefetchwt1, -march=knl, -march=knm, -mtune=knl or -mtune=knm compiler switches. Support will be removed in GCC 15.
the issue was that coordinating across this kind of hierarchy wasted a bunch of time. If you already knew how to coordinate, mostly, you could instead get better performance

you might be surprised but we're getting to the point that communicating over a super computer is on the same order of magnitude as talking across a numa node.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#350

Post by qwertox »

How come the answers for Mixtral 8x7B-32k and Llama 2 70B-4k are identical?

After asking via Mixtral a couple of questions I switched to Llama, and while it shows Llama as the Model used for the response, the answer is identical.

See first and last question:

https://pastebin.com/ZQV10C8Q

Post reply on HN