Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq runs Mixtral 8x7B-32k with 500 T/s

groq.com

Post reply on HN
Print view View original

201–210 of 482 posts

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#201

Post by tome »

vimarsh6739 wrote: post #196
tome wrote: post #75

Earlier quoted context omitted.

Yes, because we're one of the only companies whose hardware can actually support low latency! Everyone else is stuck with traditional designs and they try to make up for their high latency by batching to get higher throughput. But not all applications work with high throughput/high latency ... Low latency unlocks feeding the result of one model into the input of another model. Check out this conversational AI demo on…

Might be a bit out of context, but isn't the TPU also optimized for low latency inference? (Judging by reading the original TPU architecture paper here - https://arxiv.org/abs/1704.04760 ). If so, does Groq actually provide hardware support for LLM inference?

Jonathan Ross on that paper is Groq's founder and CEO. Groq's LPU is an natural continuation of the breakthrough ideas he had when designing Google's TPU.

Could you clarify your question about hardware support? Currently we build out our hardware to support our cloud offering, and we sell systems to enterprise customers.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#202

Post by sorokod »

Cheer2171 wrote: post #132
sorokod wrote: post #93

Earlier quoted context omitted.

Well ok, Groq provides lower latency cheaper access to the same models of questionable quality. Is this not putting lipstick on a pig scenario? I suppose more of a question to pig buyers.

Okay. How about this: Someone posts to HN about an amazing new battery technology, which they demo by showing an average-sized smartphone watching TikTok endlessly scroll for over 500 hours on a single charge. Then someone comments that TikTok is a garbage fire and a horrible corrupting influence, yadda yadda, all that stuff. They ask: what is the point of making phones last longer just to watch TikTok? They say this…

Perhaps your analogy reveals more then you intended.

What does it tell you about the new technology if the best vehicle to demonstrate it is TikTok?

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#203

Post by tome »

karthityrion wrote: post #175
tome wrote: post #5

Hi folks, I work for Groq. Feel free to ask me any questions. (If you check my HN post history you'll see I post a lot about Haskell. That's right, part of Groq's compilation pipeline is written in Haskell!)

What is the underlying architecture of the ASICs. Does it use systolic arrays?

Yes, our matrix engine is quite similar to a systolic array. You can find more details about our architecture in our paper:

https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#204

Post by frognumber »

KRAKRISMOTT wrote: post #200
317070 wrote: post #188

Earlier quoted context omitted.

Even if it isn't true. Disclosing inside information is illegal, _even if it is false and fabricated_, if it leads to personal gains.

You have to prove the OP had personal gains. If he's just a troll, it will be difficult.

You also have to be an insider.

If I go to a bar, and overhear a pair of Googlers discussing something secret and overhear it, I can:

1) Trade on it.

2) Talk about it.

Because I'm not an insider. On the other hand, if I'm sleeping with the CEO, I become an insider.

Not a lawyer. Above is not legal advice. Just a comment that the line is much more complex, and talking about a potential acquisition is usually okay (if you're not under NDA).

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#205

Post by frognumber »

tome wrote: post #91
treesciencebot wrote: post #80

The main problem with the Groq LPUs is, they don't have any HBM on them at all. Just a miniscule (230 MiB) [0] amount of ultra-fast SRAM (20x faster than HBM3, just to be clear). Which means you need ~256 LPUs (4 full server racks of compute, each unit on the rack contains 8x LPUs and there are 8x of those units on a single rack) just to serve a single model [1] where as you can get a single H200 (1/256 of the server…

If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantee…

From a theoretical perspective, this is absolutely not true. Asynchronous logic can achieve much lower latency guarantees than synchronous logic.

Come to think of it, this is one of the few places where asynchronous logic might be more than academic... Async logic is hard with complex control flows, which deep learning inference does not have.

(From a practical perspective, I know you were comparing to independently-clocked logic, rather than async logic)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#207

Post by vimarsh6739 »

tome wrote: post #201
vimarsh6739 wrote: post #196

Earlier quoted context omitted.

Might be a bit out of context, but isn't the TPU also optimized for low latency inference? (Judging by reading the original TPU architecture paper here - https://arxiv.org/abs/1704.04760 ). If so, does Groq actually provide hardware support for LLM inference?

Jonathan Ross on that paper is Groq's founder and CEO. Groq's LPU is an natural continuation of the breakthrough ideas he had when designing Google's TPU. Could you clarify your question about hardware support? Currently we build out our hardware to support our cloud offering, and we sell systems to enterprise customers.

Thanks for the quick reply! About hardware support, I was wondering if the LPU has a hardware instruction to compute the attention matrix similar to the MatrixMultiply/Convolve instruction in the TPU ISA. (Maybe a hardware instruction which fuses a softmax on the matmul epilogue?)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#208

Post by totalhack »

tome wrote: post #198
totalhack wrote: post #181

Where is the data center located? The fastest response time I could get from some quick testing from the northeast US, having it output just one letter, was 670ms. Just wondering if that's an expected result, as it's on a par or slower than GPT 3.5 via API.

West Coast US. You would have been placed in our queuing system because with all the attention we are getting we are very busy right now!

Thanks! I did notice the queue count showing up occasionally but not every time. Maybe someone could repeat the test who has access without the queue so we can get an understanding of the potential latency once scaled and geo-distributed. What I'm really trying to understand is time to first token output actually faster than GPT 3.5 via API or just the rate of token output once it begins.

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#209

Post by sebastiennight »

neilv wrote: post #176

If the page can't access certain fonts, it will fail to work, while it keeps retrying requests: https://fonts.gstatic.com/s/notosansarabic/[...] https://fonts.gstatic.com/s/notosanshebrew/[...] https://fonts.gstatic.com/s/notosanssc/[...] (I noticed this because my browser blocks these de facto trackers by default.)

Same problem when trying to use font replacements with a privacy plugin.

This is a very weird dependency to have :-)

Re: Groq runs Mixtral 8x7B-32k with 500 T/s

#210

Post by jkachmar »

yzh wrote: post #169

Really impressive work! I wonder how easy would it be to support (a future open source version of) SORA using Groq's design. Will there be a Video Processing Unit (VPU)?

i can't comment about sora specifically, however the architecture can support workloads beyond just LLM inference.

our demo booth at trade shows usually has StyleCLIP up at one point or another to provide an abstract example of this.

disclosure: i work on infrastructure at Groq and am generally interested in hardware architecture and compiler design, however i am not a part of either of those teams :)

Post reply on HN