Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

441–450 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#441

Post by frognumber »

jazzyjackson wrote: post #406
frognumber wrote: post #204

Earlier quoted context omitted.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

just so you know no one's ever been taken to court for discussing the law, it doesn't matter that you're not a lawyer, it's basically a meme

Just so you know, plenty of people have been penalized for practicing law without a license. If someone engages in insider trading based on a mistake you made on the internet, you can be liable.

In my jurisdiction, that would involve me taking money (not just talking on the internet), so I'm not at risk, but in plenty of states, you can be. A lot of this hinges on the difference between "legal information" (which is generic) and "legal advice" (which is specific).

There are whole law review articles on this, which I read more than a decade ago, nerding on something related.

But that's beside the point. A major reason for the disclaimer is that people SHOULD be aware of my level of expertise. I do the same on technical posts too. I'll disclaim whether e.g. I have world-class expertise in a topic, worked in an adjacent domain, or read a blog post somewhere (and wish others did too). It's helpful to know people's backgrounds. I am NOT a lawyer specializing in securities law. I know enough to tell people the line is more complex than trading on non-public information, but I am utterly unqualified to tell people where that line is. If you're planning to do that, you SHOULD NOT rely on it. Either read relevant case law, talk to a genuine lawyer who specializes in this stuff, or find some other way to educate yourself on whether what you're doing is okay.

So it does matter I'm not a lawyer, if not for the reasons you mentioned.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#442

Post by ponywombat »

This is very impressive, but whilst it was very fast with Mixtral yesterday, today I waited 59.44s for a response. If I was to use your API, the end-to-end is much more important than the Output Tokens Throughput and Time to first token metrics. Will you also publish average / minimum / maximum end-to-end times too?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#444

Post by xzyaoi »

londons_explore wrote: post #164
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

> more than a single model and a lot of finetunes/high rank LoRAs I can imagine a way might be found to host a base model and a bunch of LoRA's whilst using barely more ram than the base model alone. The fine-tuning could perhaps be done in such a way that only perhaps 0.1% of the weights are changed, and for every computation the difference is computed not over the weights, but of the output layer activations.

There's also papers for hosting full-parameter fine-tuned models: https://arxiv.org/abs/2312.05215

Disclaimer: I'm one of the authors.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#445

Post by SuchAnonMuchWow »

frognumber wrote: post #244
tome wrote: post #230

Earlier quoted context omitted.

Definitely, that's why we've opened our API to everyone.

I don't think that quite does it. What I'd want -- if you want me to support you -- is access to the chip, libraries, and API documentation. Best-case would be something I buy for (The mistake a lot of services make is to limit free tier to e.g. 30 day or 1 year, rather than hours/month; if I didn't get around to evaluating, switch employers, switch projects, etc. the free tier is gone). I did sign up for your API se…

The issue with their approach is that the whole LLM must fit in the chips to run at all: you need hundreds of cards to run a 7B LLM.

This approach is very good if you want to spend several millions building a large inference server to achieve the lowest latency possible. But it doesn't make sense for a lone customer buying a single card, since you wouldn't really be able to run anything on it.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#446

Post by KaiserPro »

nickpsecurity wrote: post #143
tome wrote: post #104

Earlier quoted context omitted.

We're providing by far the lowest latency LLM engine on the planet. You can't reduce latency by scaling horizontally.

Distributed, shared memory machines used to do exactly that in HPC space. They were a NUMA alternative. It works if the processing plus high-speed interconnect are collectively faster than the request rate. The 8x setups with NVLink are kind of like that model. You may have meant that nobody has a stack that uses clustering or DSM with low-latency interconnects. If so, then that might be worth developing given prior…

> Distributed, shared memory machines used to do exactly that in HPC space.

reformed HPC person here.

Yes, but not latency optimised in the case here. HPC is normally designed for throughput. Accessing memory from outside your $locality is normally horrifically expensive, so only done when you can't avoid it.

For most serving cases, you'd be much happier having a bunch of servers with a number of groqs in them, than managing a massive HPC cluster and trying to keep it both up and secure. The connection access model is much more traditional.

Shared memory clusters are not really compatible with secure enduser access. It is possible to partition memory access, but its something thats not off the shelf (well that might have changed recently.) Also, shared memory means shared fuckups.

I do get what you're hinting at, but if you want to serve low latency, high compute "messages" then discrete "APU" cards are a really good way to do it simply (assuming you can afford it). HPCs are fun, but its not fun trying to keep them up with public traffic on them

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#447

Post by tome »

ponywombat wrote: post #442

This is very impressive, but whilst it was very fast with Mixtral yesterday, today I waited 59.44s for a response. If I was to use your API, the end-to-end is much more important than the Output Tokens Throughput and Time to first token metrics. Will you also publish average / minimum / maximum end-to-end times too?

Yes, sorry about that, it's because of the huge uptick in demand we've had since we went viral. We're building out more and more hardware to cope with demand. I don't think we have any quality of service guarantees for our free tier, but you can email sales@groq.com to discuss your needs.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#448

Post by tome »

lynguist wrote: post #440
tome wrote: post #345

Earlier quoted context omitted.

Currently graphics processors work well for training. Language processors (LPUs) excel at inference.

Did you custom build those Language processors for this task? Or did you repurpose something already existing? I have never heard anyone use ‘Language processor’ before.

The chips are built for general purpose low latency, high throughput numerical compute.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#450

Post by tome »

moffkalast wrote: post #432
SeanAnderson wrote: post #220

Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…

I think NVidia is listing max throughput in terms of batching, so e.g. 50 tok/s for 10 different prompts at the same time. Groq LPUs definitely outerform an H100 in raw speed. But fundamentally it's a system that only has 10x the speed for 500x the price, made by a company that runs a blockchain and is trying to heavily market what were intended to be crypto mining chips for LLM inference. It's really quite a funny c…

Groq does not run a blockchain and our chips were never intended for crypto mining.
Post reply on HN