Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

381–390 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#381

Post by fennecbutt »

benchess wrote: post #16
WiSaGaN wrote: post #13

Earlier quoted context omitted.

How much do 568 chips cost? What’s the cost ratio of it comparing to setup with roughly the same throughput using A100?

They’re for sale on Mouser for $20625 each https://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109... At that price 568 chips would be $11.7M

I presume that's because it's a custom asic not yet in mass production?

If they can get costs down and put more dies into each card then it'll be business/consumer friendly.

Let's see if they can scale production.

Also, where tf is the next coral chip, alphabet been slacking hard.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#382

Post by foundval »

keeshond wrote: post #375

I see XTX is one of the investors - any potential use cases that require deterministic computation that you can talk about beyond just inference?

(Groq Employee) As I'm sure you're aware, XTX takes its name from a particular linear algebra operation that happens to be used a lot in Finance.

Groq happens to be excellent at doing huge linear algebra operations extremely fast. If they are latency sensitive, even better. If they are meant to run in a loop, best - that reduces the bandwidth cost of shipping data into and outside of the system. So think linear algebra driven search algorithms. ML Training isn't in this category because of the bandwidth requirements. But using ML inference to intelligently explore a search space? bingo.

If you dig around https://wow.groq.com/press, you'll find multiple such applications where we exceeded existing solutions by orders of magnitude.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#383

Post by robrenaud »

frozenport wrote: post #343
jonplackett wrote: post #340

Earlier quoted context omitted.

Is this useful for training as well as running a model. Or is this approach specifically for running an already-trained model faster?

In principle, training is basically the same as running inference but iteratively, in practice training would use a different software stack.

Training requires a lot more memory to keep gradients + gradient stats for the optimizer, and needs higher precision weights for the optimization. It's also much more parallelizable. But inference is kind of a subroutine of training.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#384

Post by fennecbutt »

Tried it out, seriously impressive. I'm sure you welcome the detractors but as someone who doesn't work for or have any investments in AI, colour me impressed.

Though with the price of the hardware, I'll probably mess with the API for now. Give us a bell when the hardware is consumer friendly, ha ha.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#385

Post by startupsfail »

matanyall wrote: post #94
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configur…

I have one, with 13B, on a 5-year-old 48GB Q8000 GPU. It’s also can see, it’s LLaVA. And it is very important that it is local, as privacy is important and streaming images to the cloud is time consuming.

You only need a few tokens, not the full 500 tokens response to run TTS. And you can pre-generate responses online, as ASR is still in progress. With a bit of clever engineering the response starts with virtually no delay, the moment its natural to start the response.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#386

Post by fennecbutt »

matanyall wrote: post #94
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configur…

Are there voice responses in the demo? I couldn't find em?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#387

Post by frognumber »

frozenport wrote: post #372
frognumber wrote: post #370

Earlier quoted context omitted.

I actually wasn't so much talking from that perspective, as simply from the perspective of the design of individual pieces. There were rather clever things done in e.g. older multipliers or adders or similar which, I think , could apply to most modern parallel architectures, be that GPGPU, SP, MPE, FPGA, or whatever, in order to significantly increase density at a cost of slightly reduced serial performance. For mach…

Truth is that there is another startup called graph core that is doing exactly that, and also a really big chip

They do what you were talking about, not what I was.

They seem annoying. "The IPU has a unique memory architecture consisting of large amounts of In-Processor-Memory™ within the IPU made up of SRAM (organised as a set of smaller independent distributed memory units) and a set of attached DRAM chips which can transfer to the In-Processor-Memory via explicit copies within the software. The memory contained in the external DRAM chips is referred to as Streaming Memory™."

There's a ™ every few words. Those seem like pretty generic terms. That's their technical documentation.

The architecture is reminiscent of some ideas from circa-2000 which didn't pan out. It reminds me of Tilera (the guy who ran it was the Donald Trump of computer architectures; company was acquihired by EZchip for a fraction of the investment which was put into it, which went to Mellanox, and then to NVidia).

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#388

Post by xanderatallah »

pests wrote: post #225
tome wrote: post #165

Earlier quoted context omitted.

You're welcome. Thanks for reporting. It's pretty confusing so maybe we should change it :)

I've always liked how openrouter.ai does it They allow you to configure chat participants (a model + params like context or temp) and then each AI answers each question independently in-line so you can compare and remix outputs.

openrouter dev here - would love to get Groq access and include it!

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#389

Post by Temporary_31337 »

pclmulqdq wrote: post #299
johntb86 wrote: post #159

Earlier quoted context omitted.

Presumably with dram you also have to worry about refreshes, which can come along at arbitrary times relative to the workload.

You can control when those happen, too.

not without affecting performance though? If you delay refreshes, this lowers performance as far as I remember...

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#390

Post by monkmartinez »

a_wild_dandan wrote: post #256
tome wrote: post #224

Earlier quoted context omitted.

We don't think so, but you be the judge! I believe we quantize both Mixtral and Llama 2 in this way.

Is your confidence rooted in quantified testing, or just vibes? I'm sure you're right, just curious. (My reasoning: running inference at full fp16 is borderline wasteful. You can use q7 with almost no loss.)

I know some fancy benchmark says "almost no loss", but... subjectively, there is a clear quality loss. You can try for yourself, I can run Mixtral at 5.8bpw and there is an OBVIOUS difference between what I have seen from Groq and my local setup beside the sound barrier shattering speed of Groq. I didn't know Mixtral could output such nice code and I have used it A LOT locally.
Post reply on HN