Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

361–370 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#361

Post by kergonath »

throwawayurlife wrote: post #265
frognumber wrote: post #204

Earlier quoted context omitted.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

It doesn't matter if you overheard it at a bar or if you're just some HN commenter posting completely incorrect legal advice; the law prohibits trading on material nonpublic information. I would pay a lot to see you try your ridiculous legal hokey-pokey on how to define an "insider."

> the law prohibits trading on material nonpublic information.

Isn’t it public information the moment it’s said audibly in a public space?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#362

Post by Aeolun »

I think we’re kind of past the point where we post prompts because it’s interesting, but this one still had me thinking.

Obviously it doesn’t have memory, but it’s the first time I’ve seen a model actually respond instead of hedge (having mostly used ChatGPT).

> what is the longest prompt you have ever received?

> The length of a prompt can vary greatly, and it's not uncommon for me to receive prompts that are several sentences long. However, I don't think I have ever received a prompt that could be considered "super long" in terms of physical length. The majority of prompts I receive are concise and to the point, typically consisting of a single sentence or a short paragraph.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#363

Post by yieldcrv »

Aeolun wrote: post #362

I think we’re kind of past the point where we post prompts because it’s interesting, but this one still had me thinking. Obviously it doesn’t have memory, but it’s the first time I’ve seen a model actually respond instead of hedge (having mostly used ChatGPT). > what is the longest prompt you have ever received? > The length of a prompt can vary greatly, and it's not uncommon for me to receive prompts that are severa…

Been using it exclusively since December, 5bit quantized, 8,000 token context window

Sometimes you need a model that just gives you the feeling “that’ll do”

I did switch to Miqu a few weeks back though. 4 bit quantized

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#364

Post by Aeolun »

tome wrote: post #227
SeanAnderson wrote: post #220

Sorry, I'm a bit naïve about all of this. Why is this impressive? Can this result not be achieved by throwing more compute at the problem to speed up responses? Isn't the fact that there is a queue when under load just indicative that there's a trade-off between "# of request to process per unit of time" and "amount of compute to put into a response to respond quicker"? https://raw.githubusercontent.com/NVIDIA/Tensor…

LLM inference is inherently a sequential problem. You can't speed it up by doing more in parallel. You can't generate the 101st token before you've generated the 100th.

They’re using several hundred cards here. Clearly there is ‘something’ that can be done in parallel.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#365

Post by Zpalmtree »

CuriouslyC wrote: post #23

This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.

GPT-4 is overpriced vs what?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#366

Post by Aeolun »

frognumber wrote: post #244
tome wrote: post #230

Earlier quoted context omitted.

Definitely, that's why we've opened our API to everyone.

I don't think that quite does it. What I'd want -- if you want me to support you -- is access to the chip, libraries, and API documentation. Best-case would be something I buy for (The mistake a lot of services make is to limit free tier to e.g. 30 day or 1 year, rather than hours/month; if I didn't get around to evaluating, switch employers, switch projects, etc. the free tier is gone). I did sign up for your API se…

I don’t really understand this. If you are happy to buy a <2K card, then what does it matter if the service is paid or not? Clearly you have enough disposable income to not care about a ‘free’ tier.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#368

Post by vitorgrs »

elorant wrote: post #274
eigenvalue wrote: post #121

I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…

Perplexity Labs also has an open demo of Mixtral 8x7b although it's nowhere near as fast as this. https://labs.perplexity.ai/

Poe has a bunch of them, including Groq as well!

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#370

Post by frognumber »

frozenport wrote: post #349
frognumber wrote: post #347

Earlier quoted context omitted.

Curiously, almost all of this video is mostly covered by computer architectures lit in the late 90's early 00's. At the time, I recall Tom Knight had done most of the analysis in this video, but I don't know if he ever published it. It was extrapolating into the distant future. To answer your questions: - Spatial processors are an insanely good fit for async logic - Matrix of processing engines are a moderately good…

This was sort of the dream of KNL but today I noticed Xeon Phi CPUs support (a.k.a. Knight Landing and Knight Mill) are marked as deprecated. GCC will emit a warning when using the -mavx5124fmaps, -mavx5124vnniw, -mavx512er, -mavx512pf, -mprefetchwt1, -march=knl, -march=knm, -mtune=knl or -mtune=knm compiler switches. Support will be removed in GCC 15. the issue was that coordinating across this kind of hierarchy was…

I actually wasn't so much talking from that perspective, as simply from the perspective of the design of individual pieces. There were rather clever things done in e.g. older multipliers or adders or similar which, I think, could apply to most modern parallel architectures, be that GPGPU, SP, MPE, FPGA, or whatever, in order to significantly increase density at a cost of slightly reduced serial performance.

For machine learning, that's a good tradeoff.

Indeed, with some of the simpler architectures, I think computation could be moved into the memory itself, as long dreamed of.

(Simply sticking 32,000 SA-110 processors on a die would be very, very limited by interconnect; there's a good reason for the types of architectures we're seeing not being that)

Post reply on HN