Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

271–280 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#271

Post by programmarchy »

throwawayurlife wrote: post #265
frognumber wrote: post #204

Earlier quoted context omitted.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."

If you did hear it in a bar, could you tweet it out before your trade, so the information is made public?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#272

Post by sofixa »

throwawayurlife wrote: post #265
frognumber wrote: post #204

Earlier quoted context omitted.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."

Had insider trading training, and yes, that's the gist of it. If you know or presume that the information is material (makes a difference) and not public, it's illegal to act on it.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#274

Post by elorant »

eigenvalue wrote: post #121

I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…

Perplexity Labs also has an open demo of Mixtral 8x7b although it's nowhere near as fast as this.

https://labs.perplexity.ai/

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#276

Post by eightysixfour »

tagyro wrote: post #275

I (only) ran a couple of prompts but I am impressed. It has the speed of gpt 3.5 and the quality of gpt 4. Seriously considering switching from [open]AI to Mix/s/tral in my apps.

Mixtral 8x7 is good, but it is not GPT-4 good in any of the use cases I have tried. Mistral’s other models get close and beat it in some cases, but not Mixtral.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#278

Post by SeanAnderson »

snowfield wrote: post #264
SeanAnderson wrote: post #236

Earlier quoted context omitted.

Yeah. That makes sense, thank you for clarifying. I updated my original post with a chart from NVIDIA which highlights the H100's capabilities. It doesn't seem unreasonable to expect a 7B model to run at 500 tok/s on that hardware.

This is a 50B model. (Mixtral 8x7b)

Oh, sorry, I assumed the 8 was for quantization. 8x7b is a new syntax for me.

Still, the NVIDIA chart shows Llama v2 70B at 750 tok/s, no?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#279

Post by mlazos »

matanyall wrote: post #94
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configur…

You can’t scale horizontally forever because of communication. I think HBM would provide a lot more flexibility with the number of chips you need.
Post reply on HN