Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

301–310 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#301

Post by nabakin »

SeanAnderson wrote: post #220

Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…

There's a difference between token throughput and latency. Token throughput is the token throughput of the whole GPU/system and latency is the token throughput for an individual user. Groq offers extremely low latency (aka extremely high token throughput per user) but we still don't have numbers on the token throughput of their entire system. Nvidia's metrics here on the other hand, show us the token throughput of the whole GPU/system. So, in reality, while you might be able to get 1.5k t/s on an H100, the latency (token throughput per user) will be something much lower like 20 t/s.

The really important metric to look for is cost per token because even though Groq is able to run at low latency, that doesn't mean it's able to do it cheaply. Determining the cost per token can be done many ways but a useful way for us is approximately the cost of the system divided by the total token throughput of the system per second. We don't have the total token throughput per second of Groq's system so we can't really say how efficient it is. It could very well be that Groq is subsidizing the cost of their system to lower prices and gain PR and will increase their prices later on.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#302

Post by johndough »

monkin wrote: post #187

It's impressive, but I have one problem with all of those models. I wanted them to answer what Mixtral or Llama2 are, but with no luck. It would be great if models could at least describe themselves.

There are two issues with that.

1. To create a model, you have to train it on training data. Mixtral and Llama2 did not exist before they were trained, so their training data did not contain any information about Mixtral or Llama2 (respectively). You could train it on fake data, but that might not work that well because:

2. The internet is full of text like "I am ", so it would probably overshadow any injected training data like "I am Llama2, a model by MetaAI."

You could of course inject the information as an invisible system prompt (like OpenAI is doing with ChatGPT), but that is a waste of computation resources.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#303

Post by johndough »

tome wrote: post #162
jart wrote: post #149

Earlier quoted context omitted.

If I understand correctly, you're using specialized hardware to improve token generation speed, which is very latency bound on the speed of computation. However generating tokens only requires multiplying 1-dimensional matrices usually. If I enter a prompt with ~100 tokens then your service goes much slower. Probably because you have to multiply 2-dimensional matrices. What are you doing to improve the computation sp…

I don't think it should be quadratic in input length. Why do you think it is?

You can ask your website: "What is the computational complexity of self-attention with respect to input sequence length?"

It'll answer something along the lines of self-attention being O(n^2) (where n is the sequence length) because you have to compute an attention matrix of size n^2.

There are other attention mechanisms with better computational complexity, but they usually result in worse large language models. To answer jart: We'll have to wait until someone finds a good linear attention mechanism and then wait some more until someone trains a huge model with it (not Groq, they only do inference).

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#304

Post by larodi »

eigenvalue wrote: post #121

I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…

why sell? it would be much more delightful to beat them on their own game?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#305

Post by kcorbitt »

londons_explore wrote: post #164
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

> more than a single model and a lot of finetunes/high rank LoRAs I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using barely more ram than the base model alone. The fine-tuning could perhaps be done in such a way that only perhaps 0.1% of the weights are changed, and for every computation the difference is computed not over the weights, but of the output layer activations.

This actually already exists! We did a writeup of the relevant optimizations here: https://openpipe.ai/blog/s-lora

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#306

Post by frozenport »

nabakin wrote: post #301
SeanAnderson wrote: post #220

Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…

There's a difference between token throughput and latency. Token throughput is the token throughput of the whole GPU/system and latency is the token throughput for an individual user. Groq offers extremely low latency (aka extremely high token throughput per user) but we still don't have numbers on the token throughput of their entire system. Nvidia's metrics here on the other hand, show us the token throughput of th…

https://wow.groq.com/artificialanalysis-ai-llm-benchmark-dou...

Seems to have it. Looks cost competitive but a lot faster.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#307

Post by NorwegianDude »

tome wrote: post #227
SeanAnderson wrote: post #220

Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…

LLM inference is inherently a sequential problem. You can't speed it up by doing more in parallel. You can't generate the 101st token before you've generated the 100th.

Technically, I guess you can use speculative execution to speed it up, and in that way take a guess at what the 100th token will be and start on the 101st token at the same time? Though it probably has it's own unforeseen challenges.

Everything is predictable with enough guesses.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#308

Post by foundval »

frognumber wrote: post #205
tome wrote: post #91

Earlier quoted context omitted.

If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantee…

From a theoretical perspective, this is absolutely not true. Asynchronous logic can achieve much lower latency guarantees than synchronous logic. Come to think of it, this is one of the few places where asynchronous logic might be more than academic... Async logic is hard with complex control flows, which deep learning inference does not have. (From a practical perspective, I know you were comparing to independently-…

(Groq Employee) You're right - we are comparing to independently-clocked logic.

I wonder whether async logic would be feasible for reconfigurable "Spatial Processor" type architectures [1]. As far as LPU architectures go, they fall in the "Matrix of Processing Engines"[1] family of architectures, which I would naively guess is not the best suited to leverage async logic.

1: I'm using the "Spatial Processor" (7:14) and "Matrix of Processing Engines" (8:57) terms as defined in https://www.youtube.com/watch?v=LUPWZ-LC0XE. Sorry for a video link, I just can't think of another single reference that explains the two approaches.

Post reply on HN