Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

191–200 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#191

Post by emporas »

CuriouslyC wrote: post #23

This is pretty sweet. The speed is nice but what I really care about is you bringing the per token cost down compared with models on the level of mistral medium/gpt4. GPT3.5 is pretty close in terms of cost/token but the quality isn't there and GPT4 is overpriced. Having GPT4 quality at sub-gpt3.5 prices will enable a lot of things though.

Mixtral's quality is definitely up there with Gpt3.5. Specifically for coding, i consider them almost equivalent in quality. In fact Mixtral 8x7 is starting to be my go-to coding assistant instead of Gpt. It is fast, it is accurate, and i think i like his responses better than Gpt.

Reducing LLM size almost 10 times in the span of a little more than a year, that's great stuff. Next step i think is 3 billion parameters MoE with 20 experts.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#192

Post by jart »

tome wrote: post #162
jart wrote: post #149

Earlier quoted context omitted.

If I understand correctly, you're using specialized hardware to improve token generation speed, which is very latency bound on the speed of computation. However generating tokens only requires multiplying 1-dimensional matrices usually. If I enter a prompt with ~100 tokens then your service goes much slower. Probably because you have to multiply 2-dimensional matrices. What are you doing to improve the computation sp…

I don't think it should be quadratic in input length. Why do you think it is?

all I know is that when I run llama.cpp a lot of the matrices that get multiplied have their shapes defined by how many tokens are in my prompt. https://justine.lol/tmp/shapes.png Notice how the B matrix is always skinny for generating tokens. But for batch processing of the initial prompt, it's fat. It's not very hard to multiply a skinny matrix but once it's fat it gets harder. Handling the initial batch processing of the prompt appears to be what your service goes slow at.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#193

Post by trsohmers »

tome wrote: post #186
trsohmers wrote: post #113

Earlier quoted context omitted.

Groq states in this article [0] that they used 576 chips to achieve these results, and continuing with your analysis, you also need to factor in that for each additional user you want to have requires a separate KV cache, which can add multiple more gigabytes per user. My professional independent observer opinion (not based on my 2 years of working at Groq) would have me assume that their COGS to achieve these perfor…

Thomas, I think for full disclosure you should also state that you left Groq to start a competitor (a competitor which doesn't have the world's lowest latency LLM engine nor a guarantee to match the cheapest per token prices, like Groq does.). Anyone with a serious interest in the total cost of ownership of Groq's system is welcome to email contact@groq.com.

I thought that was clear through my profile, but yes, Positron AI is focused on providing the best performance per dollar while providing the best quality of service and capabilities rather than just focusing on a single metric of speed.

A guarantee to match the cheapest per token prices is sure a great way to lose a race to the bottom, but I do wish Groq (and everyone else trying to compete against NVIDIA) the greatest luck and success. I really do think that the great single batch/user performance by Groq is a great demo, but is not the best solution for a wide variety of applications, but I hope it can find its niche.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#195

Post by tome »

WanderPanda wrote: post #185
tome wrote: post #158

Earlier quoted context omitted.

I think existing players will have trouble developing a low latency solution like us whilst they are still running on non-deterministic hardware.

What do you mean by non-deterministic hardware? cuBLAS on a laptop GPU was deterministic when I tried it last iirc

Non-deterministic timing characteristics.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#196

Post by vimarsh6739 »

tome wrote: post #75
mechagodzilla wrote: post #24

Earlier quoted context omitted.

You all seem like one of the only companies targeting low-latency inference rather than focusing on throughput (and thus $/inference) - what do you see as your primary market?

Yes, because we're one of the only companies whose hardware can actually support low latency! Everyone else is stuck with traditional designs and they try to make up for their high latency by batching to get higher throughput. But not all applications work with high throughput/high latency ... Low latency unlocks feeding the result of one model into the input of another model. Check out this conversational AI demo on…

Might be a bit out of context, but isn't the TPU also optimized for low latency inference? (Judging by reading the original TPU architecture paper here - https://arxiv.org/abs/1704.04760). If so, does Groq actually provide hardware support for LLM inference?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#197

Post by frozenport »

WanderPanda wrote: post #185
tome wrote: post #158

Earlier quoted context omitted.

I think existing players will have trouble developing a low latency solution like us whilst they are still running on non-deterministic hardware.

What do you mean by non-deterministic hardware? cuBLAS on a laptop GPU was deterministic when I tried it last iirc

Tip of the ice-berg.

DRAM needs to be refreshed every X cycles.

This means you don't know the time it takes to read from memory. You could be reading at a refresh cycle. This circuitry also adds latency.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#198

Post by tome »

totalhack wrote: post #181

Where is the data center located? The fastest response time I could get from some quick testing from the northeast US, having it output just one letter, was 670ms. Just wondering if that's an expected result, as it's on a par or slower than GPT 3.5 via API.

West Coast US. You would have been placed in our queuing system because with all the attention we are getting we are very busy right now!

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#199

Post by Klaus23 »

The demo is pretty cool, but the mobile interface could be a parody of bad interface design. The text box at the top is hard to reach if you want to open the keyboard, which automatically closes, or press the button to send the question, and the chat history is out of chronological order for no logical reason.

Edit: Text selection is also broken.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#200

Post by KRAKRISMOTT »

317070 wrote: post #188
baq wrote: post #183

Earlier quoted context omitted.

If this is true, expect a call from the SEC...

Even if it isn't true. Disclosing inside information is illegal, _even if it is false and fabricated_, if it leads to personal gains.

You have to prove the OP had personal gains. If he's just a troll, it will be difficult.
Post reply on HN