Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

131–140 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#131

Post by siwakotisaurav »

sorokod wrote: post #93
Cheer2171 wrote: post #79

Earlier quoted context omitted.

No, it is "do whatever you were already doing with ML, faster" This question seems either from a place of deep confusion or is in bad faith. This post is about hardware. The hardware is model independent.* Any issues with models, like hallucinations, are going to be identical if it is run on this platform or a bunch of Nvidia GPUs. Performance in terms of hardware speed and efficiency are orthogonal to performance in…

Well ok, Groq provides lower latency cheaper access to the same models of questionable quality. Is this not putting lipstick on a pig scenario? I suppose more of a question to pig buyers.

They’re probably in the business of being the hardware provider. Best thing would be if Microsoft buys a lot of their chips and that way chatgpt is actually sped up. It’s basically model independent

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#132

Post by Cheer2171 »

sorokod wrote: post #93
Cheer2171 wrote: post #79

Earlier quoted context omitted.

No, it is "do whatever you were already doing with ML, faster" This question seems either from a place of deep confusion or is in bad faith. This post is about hardware. The hardware is model independent.* Any issues with models, like hallucinations, are going to be identical if it is run on this platform or a bunch of Nvidia GPUs. Performance in terms of hardware speed and efficiency are orthogonal to performance in…

Well ok, Groq provides lower latency cheaper access to the same models of questionable quality. Is this not putting lipstick on a pig scenario? I suppose more of a question to pig buyers.

Okay. How about this: Someone posts to HN about an amazing new battery technology, which they demo by showing an average-sized smartphone watching TikTok endlessly scroll for over 500 hours on a single charge.

Then someone comments that TikTok is a garbage fire and a horrible corrupting influence, yadda yadda, all that stuff. They ask: what is the point of making phones last longer just to watch TikTok? They say this improved efficiency in battery tech is just putting lipstick on a pig.

That's you in this thread. That's the kind of irrelevant non-contribution you are making here.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#133

Post by patapong »

siwakotisaurav wrote: post #126
patapong wrote: post #112

Very impressive! I am even more impressed by the API pricing though - 0.27/1M tokens seems like an order of magnitude cheaper than the GPT-3.5 API, and two orders of magnitude cheaper than GPT-4? Am I missing something here?

They’re competing with the lowest cost competitors for mistral atm, which afaik is currently deepinfra at the same pricing

Huh! Had no idea open source model were ahead of OpenAI already on pricing - will have to look into using these for my use cases.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#135

Post by mrtksn »

SahAssar wrote: post #129
mrtksn wrote: post #118

Earlier quoted context omitted.

Yes, IIRC they reason on the car actions using LLMs. They still use image processing but once you identify the objects in the scene, the LLM interprets and decides what to do with the car. I'm not sure which one was it though(Ghost Autonomy maybe?).

Do you have a source? Because that actually, properly working would be headline global news and would value the company in the billions.

It was discussed here on HN, that's how I know about it.

I found a few things when searched around but not sure which one was the one I recall.

Anyway, here is a video from one: https://www.youtube.com/watch?v=C2rbym6bXM0

Here is a paper discussing something similar: https://arxiv.org/abs/2307.07162

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#136

Post by losvedir »

karpathy wrote: post #59

Very impressive looking! Just wanted to caution it's worth being a bit skeptical without benchmarks as there are a number of ways to cut corners. One prominent example is heavy model quantization, which speeds up the model at a cost of model quality. Otherwise I'd love to see LLM tok/s progress exactly like CPU instructions/s did a few decades ago.

Maybe I'm stretching the analogy too far, but are we in the transistor regime of LLMs already? Sometimes I see these 70 billion parameter monstrosities and think we're still building ENIAC out of vacuum tubes.

In other words, are we ready to steadily march on, improving LLM tok/s year by year, or are we a major breakthrough or two away before that can even happen?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#137

Post by eurekin »

Jaw dropping. Both groq and mixtral.

I used following prompt:

Generate gitlab ci yaml file for a hybrid front-end/backend project. Fronted is under /frontend and is a node project, packaged with yarn, built with vite to the /backend/public folder. The backend is a python flask server

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#138

Post by tome »

tome wrote: post #90
sva_ wrote: post #65

In how far is the API compatible with OpenAI? Does it offer logprobs[0] and top_logprobs[1]? 0. https://platform.openai.com/docs/api-reference/chat/create#c... 1. https://platform.openai.com/docs/api-reference/chat/create#c...

You can find our API docs here, including details of our OpenAI compatibility https://docs.api.groq.com/

By the way, we also have a new Discord server where we are hosting our developer community. If you find anything missing in our API you can ask about there:

https://discord.com/invite/TQcy5EBdCP

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#139

Post by tome »

eigenvalue wrote: post #121

I just want to say that this is one of the most impressive tech demos I’ve ever seen in my life, and I love that it’s truly an open demo that anyone can try without even signing up for an account or anything like that. It’s surreal to see the thing spitting out tokens at such a crazy rate when you’re used to watching them generate at one less than one fifth that speed. I’m surprised you guys haven’t been swallowed up…

Really glad you like it! We've been working hard on it.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#140

Post by SahAssar »

mrtksn wrote: post #135
SahAssar wrote: post #129

Earlier quoted context omitted.

Do you have a source? Because that actually, properly working would be headline global news and would value the company in the billions.

It was discussed here on HN, that's how I know about it. I found a few things when searched around but not sure which one was the one I recall. Anyway, here is a video from one: https://www.youtube.com/watch?v=C2rbym6bXM0 Here is a paper discussing something similar: https://arxiv.org/abs/2307.07162

The description for that video says

> Ghost Autonomy’s MLLM-based capabilities are currently in development. These video and image examples show MLLM-based analysis of driving scenes captured from Ghost vehicles driving in both autonomous and conventional mode. MLLM-based reasoning is not yet being returned to the car to impact actual driving maneuvers.

So the model discussed is not doing any driving whatsoever. This is not self-driving at any level.

Post reply on HN