Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

401–410 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#401

Post by int_19h »

fragmede wrote: post #124
tome wrote: post #17

Earlier quoted context omitted.

Yeah, it's nothing to do with Elon and we (Groq) had the name first. It's a natural choice of name for something in the field of AI because of the connections to the hacker ethos, but we have the trademark and Elon doesn't. https://wow.groq.com/hey-elon-its-time-to-cease-de-grok/

I mean it sucks that Elon went and claimed Grok when you want Groq, plus you were there first, but getting stuck on the name seems like it's going to be a distraction, so why not choose something different? When Grok eventually makes the news for some negative thing, so you really want that erroneously associated with your product? Do you really want to pick a fight with the billionaire that owns Twitter, is that a c…

If anything, getting in a very public fight with Musk may well be beneficial wrt brand recognition. Especially if he responds in his usual douchy way and it gets framed accordingly in the media.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#402

Post by int_19h »

CuriouslyC wrote: post #23

This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.

You seem to be implying that Mistral Medium is on the same level as GPT-4?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#403

Post by nickpsecurity »

trsohmers wrote: post #113
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

Groq states in this article [0] that they used 576 chips to achieve these results, and continuing with your analysis, you also need to factor in that for each additional user you want to have requires a separate KV cache, which can add multiple more gigabytes per user. My professional independent observer opinion (not based on my 2 years of working at Groq) would have me assume that their COGS to achieve these perfor…

What happened to Rex? Did it hit production or get abandoned?

It was also on my list of things to consider modifying for an AI accelerator. :)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#404

Post by fragmede »

int_19h wrote: post #401
fragmede wrote: post #124

Earlier quoted context omitted.

I mean it sucks that Elon went and claimed Grok when you want Groq, plus you were there first, but getting stuck on the name seems like it's going to be a distraction, so why not choose something different? When Grok eventually makes the news for some negative thing, so you really want that erroneously associated with your product? Do you really want to pick a fight with the billionaire that owns Twitter, is that a c…

If anything, getting in a very public fight with Musk may well be beneficial wrt brand recognition. Especially if he responds in his usual douchy way and it gets framed accordingly in the media.

it'll be great for publicity, for sure

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#405

Post by jasonjmcghee »

chasd00 wrote: post #393
qeternity wrote: post #360

Earlier quoted context omitted.

It’s important…if you’re building a chatbot. The most interesting applications of LLMs are not chatbots.

> The most interesting applications of LLMs are not chatbots. What are they then? Every use case I’ve seen is either a chatbot or like a copy editor which is just a long form chatbot.

Obviously not op, but these days LLMs can be fuzzy functions with reliably structured output, and are multi-modal.

Think about the implications of that. I bet you can come up with some pretty cool use cases that don't involve you talking to something over chat.

One example:

I think we'll be seeing a lot of "general detectors" soon. Without training or predefined categories, get pinged when (whatever you specify) happens. Whether it's a security camera, web search, event data, etc

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#406

Post by jazzyjackson »

frognumber wrote: post #204
KRAKRISMOTT wrote: post #200

Earlier quoted context omitted.

You have to prove the OP had personal gains. If he's just a troll, it will be difficult.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

just so you know no one's ever been taken to court for discussing the law, it doesn't matter that you're not a lawyer, it's basically a meme

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#407

Post by nycdatasci »

chasd00 wrote: post #393
qeternity wrote: post #360

Earlier quoted context omitted.

It’s important…if you’re building a chatbot. The most interesting applications of LLMs are not chatbots.

> The most interesting applications of LLMs are not chatbots. What are they then? Every use case I’ve seen is either a chatbot or like a copy editor which is just a long form chatbot.

Complex data tagging/enrichment tasks.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#408

Post by uptownfunk »

It is fast, but if it spits useless garbage, then useless. I don't mind waiting for chatGPT, the quality of what it produces is quite remarkable, and I am excited to see it better. I think this has more to do with mistral model v GPT4 than Groq. If Groq can host GPT4, wow, then that is amazing.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#409

Post by p1esk »

jazzyjackson wrote: post #406
frognumber wrote: post #204

Earlier quoted context omitted.

You also have to be an insider. If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can: 1) Trade on it. 2) Talk about it. Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider. Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if…

just so you know no one's ever been taken to court for discussing the law, it doesn't matter that you're not a lawyer, it's basically a meme

Are you a lawyer?
Post reply on HN