Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

231–240 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#231

Post by dmw_ng »

imiric wrote: post #111

Impressive demo! However, the hardware requirements and cost make this inaccessible for anyone but large companies. When do you envision that the price could be affordable for hobbyists? Also, while the CNN Vapi demo was impressive as well, a few weeks ago here[1] someone shared https://smarterchild.chat/ . That also has _very_ low audio latency, making natural conversation possible. From that discussion it seems tha…

Given the size of the Sindarin team (3 AFAICT), that mostly looks like a clever combination of existing tech. There are some speech APIs that offer word-by-word realtime transcription (Google has one), assuming most of the special sauce is very well thought out pipelining between speech recognition->LLM->TTS

(not to denigrate their awesome achievement, I would not be interested if I were not curious about how to reproduce their result!)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#233

Post by tome »

varunvummadi wrote: post #219
tome wrote: post #147

Earlier quoted context omitted.

Definitely not, but even with a comparison to 500 GPUs Groq will still come out on top because you can never reduce latency by adding more parallel compute :)

So please let me know if I am wrong are you guys running a batch size of 1 in 500 GPU's? then why are the responses almost instant if you guys are using batch size 1 and also when can we expect bring your own fine tuned models kind of thing. Thanks!

We are not using 500 GPUs, we are using a large system built from many of our own custom ASICs. This allows us to do batch size 1 with no reduction in overall throughput. (We are doing pipelining though, so many users are using the same system at once).

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#236

Post by SeanAnderson »

MasterScrat wrote: post #226
SeanAnderson wrote: post #220

Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…

Scaling up compute can improve throughput, but can't easily improve latency between tokens. Generation is usually bottlenecked by the time it takes to go through the network for each token. To speed that up, you need to perform these computations faster, which is a hard problem after you've exhausted all the obvious options (use the fastest accelerator you can find, cache what you can etc).

Yeah. That makes sense, thank you for clarifying. I updated my original post with a chart from NVIDIA which highlights the H100's capabilities. It doesn't seem unreasonable to expect a 7B model to run at 500 tok/s on that hardware.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#237

Post by neonbrain »

idiotlogical wrote: post #151

I asked it "What is the oldest known written recipe" and then asked for that recipe. It seemed to repeat the same response a bunch of times (very quickly!) before stopping. Is this abnormal behavior?

For me the reply for continuation question "Please tell me the exact steps of that recipe.":

Here is a rough translation of the oldest known written recipe, which is for a kind of beer from ancient Mesopotamia:

1. Crush and mash 4 liters of barley to create malt.

2. Soak the malt in water for 2.5 days.

3. Leave the mash in a cool place for 2.5 days.

4. Filter the mash through a strainer to separate the liquid from the solids.

5. Boil the liquid for 1 hour.

6. Add 1 liter of honey and 1 handful of hops to the liquid.

7. Leave the mixture to cool.

8. Add 1 handful of "gig" (a type of wild yeast) to the mixture.

9. Leave the mixture in a cool place for 5 days to allow fermentation.

10. Filter the mixture again to separate the beer from the solids.

11. Serve the beer chilled.

It's important to note that this recipe is over 3,700 years old, and it may not produce a beer that is palatable to modern tastes. Additionally, some of the ingredients and techniques used in this recipe may not be readily available or practical for home brewers today.

Post reply on HN