Groq CEO: 'We No Longer Sell Hardware'

#1

Post by frozenport » Sun, Apr 07, 2024, 10:40 PM UTC

Groq CEO: 'We No Longer Sell Hardware'
eetimes.com

Re: Groq CEO: 'We No Longer Sell Hardware'

#2

Post by BoorishBears » Sun, Apr 07, 2024, 11:35 PM UTC

Read: We're forcing someone's hand in acquiring us.

Groq is still under a 30 request per minute rate-limit, which drops to 10 requests per minute if you have all day usage.

Billing has been "coming soon" this whole time, and while they've built out hype enabling features like function calling, somehow they can't setup a Stripe webhook to collect money for realistic rate limits.

They couldn't scream "we can't service the tiniest bit of our demand" any louder at this point.

_

Edit: For anyone looking for fast inference without the smoke and mirrors, I've been using Fireworks.ai in production and it's great. 200 tk/s - 300 tk/s is closer to Groq than it is to OpenAI and co.

And as a bonus they support PEFT with serverless pricing.

Re: Groq CEO: 'We No Longer Sell Hardware'

#3

Post by scosman » Sun, Apr 07, 2024, 11:37 PM UTC

Wildly fast inference. And current chips are 14nm so headroom to get a lot better.

Re: Groq CEO: 'We No Longer Sell Hardware'

#4

Post by arthurcolle » Sun, Apr 07, 2024, 11:47 PM UTC

BoorishBears wrote: post #2

Read: We're forcing someone's hand in acquiring us. Groq is still under a 30 request per minute rate-limit, which drops to 10 requests per minute if you have all day usage. Billing has been "coming soon" this whole time, and while they've built out hype enabling features like function calling, somehow they can't setup a Stripe webhook to collect money for realistic rate limits. They couldn't scream "we can't service…

they don't even let us pay them, it's insane

I just have free API access with no ability to add a credit card.

Re: Groq CEO: 'We No Longer Sell Hardware'

#5

Post by [deleted] » Sun, Apr 07, 2024, 11:47 PM UTC

BoorishBears wrote: post #2

Read: We're forcing someone's hand in acquiring us. Groq is still under a 30 request per minute rate-limit, which drops to 10 requests per minute if you have all day usage. Billing has been "coming soon" this whole time, and while they've built out hype enabling features like function calling, somehow they can't setup a Stripe webhook to collect money for realistic rate limits. They couldn't scream "we can't service…

[deleted]

Re: Groq CEO: 'We No Longer Sell Hardware'

#6

Post by gandalfgeek » Sun, Apr 07, 2024, 11:47 PM UTC

They're calling the lie on needing bleeding edge hardware for performance.

5 yr old silicon (14 nm!!) and no hbm.

Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism.

https://wow.groq.com/isca-2022-paper

Re: Groq CEO: 'We No Longer Sell Hardware'

#7

Post by brcmthrowaway » Sun, Apr 07, 2024, 11:48 PM UTC

BoorishBears wrote: post #2

Read: We're forcing someone's hand in acquiring us. Groq is still under a 30 request per minute rate-limit, which drops to 10 requests per minute if you have all day usage. Billing has been "coming soon" this whole time, and while they've built out hype enabling features like function calling, somehow they can't setup a Stripe webhook to collect money for realistic rate limits. They couldn't scream "we can't service…

What are you using all this for? Whats the product?

Re: Groq CEO: 'We No Longer Sell Hardware'

#8

Post by halflings » Sun, Apr 07, 2024, 11:51 PM UTC

gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

No HBM because they use tons of fast SRAM instead. Isn't that the main driver for performance here?

(the way I understood it => it's still cost effective at scale due to throughput increase this brings)

Re: Groq CEO: 'We No Longer Sell Hardware'

#9

Post by dsrtslnd23 » Sun, Apr 07, 2024, 11:51 PM UTC

So unless there are new Croq datacenters coming, this is only interesting for North American users. Otherwise H100 based latency optimized solutions would be faster - in particular for time-to-first-token sensitive applications.

Re: Groq CEO: 'We No Longer Sell Hardware'

#10

Post by dralley » Sun, Apr 07, 2024, 11:53 PM UTC

gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

So Itanium and its "sufficiently smart compiler" but functional?