Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

91–100 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#91

Post by tome »

treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantees disappear very quickly.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#92

Post by tome »

UncleOxidant wrote: post #83
tome wrote: post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

When will we be able to buy Groq accelerator cards that would be affordable for hobbyists?

We are prioritising building out whole systems at the moment I don't think we'll have a consumer level offering in the near future.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#93

Post by sorokod »

Cheer2171 wrote: post #79
MuffinFlavored wrote: post #58

Earlier quoted context omitted.

But if the model is useless/full of hallucinations, why does the speed of its output matter? "generate hallucinated results, faster"

No, it is "do whatever you were already doing with ML, faster" This question seems either from a place of deep confusion or is in bad faith. This post is about hardware. The hardware is model independent.* Any issues with models, like hallucinations, are going to be identical if it is run on this platform or a bunch of Nvidia GPUs. Performance in terms of hardware speed and efficiency are orthogonal to performance in…

Well ok, Groq provides lower latency cheaper access to the same models of questionable quality.

Is this not putting lipstick on a pig scenario? I suppose more of a question to pig buyers.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#94

Post by matanyall »

treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configuration. Does that mean that factories are inefficient?

You also say that H200's work reasonably well, and that's reasonable (but debatable) for synchronous, human interaction use cases. Show me a 30b+ parameter model doing RAG as part of a conversation with voice responses in less than a second, running on Nvidia.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#95

Post by wmf »

mrtksn wrote: post #89
wmf wrote: post #87

Earlier quoted context omitted.

The thing is, I don't see any efficiency improvements. I see models running fast on very expensive hardware using techniques that don't scale down.

Care to explain? Are they using 10x energy for 10x speed improvements?

They're using hundreds of chips. Based on the data sheet I would estimate this demo uses 173 KW. It may be 100x energy to get 10x speedup.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#96

Post by tome »

itishappy wrote: post #40
tome wrote: post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

Alright, I'll bite. Haskell seems pretty unique in the ML space! Any unique benefits to this decision, and would you recommend it for others? What areas of your project do/don't use Haskell?

Haskell is a great language for writing compilers! The end of our compilation pipeline is written in Haskell. Other stages are written in C++ (MLIR) and Python. I'd recommend anyone to look at Haskell if they have a compiler-shaped problem, for sure.

We also use Haskell on our infra team. Most of our CI infra is written in Haskell and Nix. Some of the chip itself was designed in Haskell (or maybe Bluespec, a Haskell-like language for chip design, I'm not sure).

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#97

Post by mrtksn »

wmf wrote: post #95
mrtksn wrote: post #89

Earlier quoted context omitted.

Care to explain? Are they using 10x energy for 10x speed improvements?

They're using hundreds of chips. Based on the data sheet I would estimate this demo uses 173 KW. It may be 100x energy to get 10x speedup.

100s of chips for who knows how many clients. The mobile phone will have to do calculations just for 1 client.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#98

Post by tome »

kkzz99 wrote: post #88
tome wrote: post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

How does the Groq PCIE Card work exactly? Does it use system ram to stream the model data to the card? How many T/s could one expect with e.g. 36000Mhz DDR4 Ram?

We build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits of low latency.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#99

Post by behnamoh »

karpathy wrote: post #59

Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.

tangent: Great to see you again on HN!

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#100

Post by pbalcer »

matanyall wrote: post #94
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq Engineer here, I'm not seeing why being able to scale compute outside of a single card/node is somehow a problem. My preferred analogy is to a car factory: Yes, you could build a car with say only one or two drills, but a modern automated factory has hundreds of drills! With a single drill, you could probably build all sorts of cars, but a factory assembly line is only able to make specific cars in that configur…

Just curious, how does this work out in terms of TCO (even assuming the price of a Groq LPU is 0$)? What you say makes sense, but I'm wondering how you strike a balance between massive horizontal scaling vs vertical scaling. Sometimes (quite often in my experience) having a few beefy servers is much simpler/cheaper/faster than scaling horizontally across many small nodes.

Or I got this completely wrong, and your solution enables use-cases that are simply unattainable on mainstream (Nvidia/AMD) hardware, making TCO argument less relevant?

Post reply on HN