Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq CEO: 'We No Longer Sell Hardware'

eetimes.com

Post reply on HN
Print view View original

31–40 of 152 posts

Re: Groq CEO: 'We No Longer Sell Hardware'

#31

Post by aleph_minus_one »

creato wrote: post #22
theturtletalks wrote: post #17

Earlier quoted context omitted.

If you read on, Groq said they would only sell hardware to US companies and outside companies would get cloud services, not the LPUs. I think the US government told them to keep the LPUs in-house since they could be the secret sauce for scale.

I'm not questioning the deployment strategy, I'm wondering why Saudi Aramco wants to access so much compute power that is highly specialized(?) for generative AI workloads. Or is it more general than that?

> I'm wondering why Saudi Aramco wants to access so much compute power that is highly specialized(?) for generative AI workloads. Or is it more general than that?

For vanity reasons and because AI is the future (not every company acts that rationally for huge buying decisions).

Re: Groq CEO: 'We No Longer Sell Hardware'

#32

Post by theturtletalks »

creato wrote: post #22
theturtletalks wrote: post #17

Earlier quoted context omitted.

If you read on, Groq said they would only sell hardware to US companies and outside companies would get cloud services, not the LPUs. I think the US government told them to keep the LPUs in-house since they could be the secret sauce for scale.

I'm not questioning the deployment strategy, I'm wondering why Saudi Aramco wants to access so much compute power that is highly specialized(?) for generative AI workloads. Or is it more general than that?

Maybe Saudi doesn’t want to rely on OpenAI and other APIs and wants to run a fine-tuned Mixtral model on the cloud or their hardware. International companies will probably opt for an open source model since the data is sensitive and OpenAI could pass that to intelligence.

Re: Groq CEO: 'We No Longer Sell Hardware'

#33

Post by themoonisachees »

aleph_minus_one wrote: post #27
latchkey wrote: post #14

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…

> If you have to sell the hardware and support people buying it, that is a world of trouble. What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?

A lot. It's the same reason amazon doesn't sell servers and instead gives you access to a single instance that everyone pretends is the same but in reality is massively transient.

Re: Groq CEO: 'We No Longer Sell Hardware'

#34

Post by latchkey »

aleph_minus_one wrote: post #27
latchkey wrote: post #14

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…

> If you have to sell the hardware and support people buying it, that is a world of trouble. What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?

Good question. Not much. If anything, what I'm doing is even harder because I will have multiple sources for the hardware. I have to deal with all of the hardware and data center issues, as well as the customers who rely on us to provide them access.

Good thing that I'm a glutton for punishment.

Re: Groq CEO: 'We No Longer Sell Hardware'

#35

Post by alted »

Custom state-of-the-art silicon is ridiculously expensive.

For a minimum 100 wafers = 10k chips, Groq may have paid $100M = $10k/chip purely in amortizing design costs.

Chip design (software + engineer time) and fabrication setup (lithography masks) grow exponentially [1][2] with smaller nodes, e.g., maybe $100M for Groq's current 14nm chips to ~$500M for their planned 4nm tapeout. Once you reach mass production (>>1000 wafers, which have ~150 large chips each), wafers are $10k each. On top of this, it takes ~1 year to design then have prototypes made. (These same issues still exist on older slower nodes, albeit not as bad.)

This could be reduced somewhat if chip design software were cheaper and margins were lower, but maybe 20% of this cost is due to fundamental manufacturing difficulty.

(disclosure: I don't work with recent tech nodes myself; this is my best guess)

[1] https://www.semianalysis.com/p/the-dark-side-of-the-semicond... [2] https://www.extremetech.com/computing/272096-3nm-process-nod...

Re: Groq CEO: 'We No Longer Sell Hardware'

#36

Post by alephnerd »

aleph_minus_one wrote: post #27
latchkey wrote: post #14

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…

> If you have to sell the hardware and support people buying it, that is a world of trouble. What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?

> What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription

Margins.

Pricing for cloud compute is much higher and servicing and management for the provider is much cheaper.

If I sold hardware directly, then I'm often on the hook for support contracts which can get pricy with hardware and distract from shipping future facing product features, as customers who purchase directly have longer upgrade windows due to logistical overhead.

Re: Groq CEO: 'We No Longer Sell Hardware'

#37

Post by latchkey »

alephnerd wrote: post #36
aleph_minus_one wrote: post #27

Earlier quoted context omitted.

> If you have to sell the hardware and support people buying it, that is a world of trouble. What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?

> What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription Margins. Pricing for cloud compute is much higher and servicing and management for the provider is much cheaper. If I sold hardware directly, then I'm often on the hook for support contracts which can get pricy with hardware and distract from shipping future facing product features, as customers…

The pricing isn't necessarily much higher. The pricing is set up to amortize the cost of running things over a period of time. If you're only going to use it for a short time, you pay more than someone who commits to a 3 year contract.

It also isn't just the hardware capex, it is everything involved under the covers. Market pricing also factors into that as well. This is something I've struggled with myself quite a bit. I know all of my costs and what I'd like to charge, but because my offering is so brand new, until my competitors announced their pricing, I wasn't sure what the market would tolerate.

Selling hardware directly is hard for exactly what you state though. Service contracts are a pain in the butt. All of this latest AI hardware has high rates of failures too. Up until recently with AMD coming to market with a great offering, the only thing people want any sort of quantity on are nvidia products. Groq probably realized that people buying 1-2 cards at a time, wasn't going to be profitable.

Re: Groq CEO: 'We No Longer Sell Hardware'

#38

Post by alephnerd »

latchkey wrote: post #37
alephnerd wrote: post #36

Earlier quoted context omitted.

> What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription Margins. Pricing for cloud compute is much higher and servicing and management for the provider is much cheaper. If I sold hardware directly, then I'm often on the hook for support contracts which can get pricy with hardware and distract from shipping future facing product features, as customers…

The pricing isn't necessarily much higher. The pricing is set up to amortize the cost of running things over a period of time. If you're only going to use it for a short time, you pay more than someone who commits to a 3 year contract. It also isn't just the hardware capex, it is everything involved under the covers. Market pricing also factors into that as well. This is something I've struggled with myself quite a b…

> it is everything involved under the covers

Yep! You explained it better than me!

> Groq probably realized that people buying 1-2 cards at a time, wasn't going to be profitable

Yep! And if would have bogged them down by slowing down R&D cycles and even fulfilling orders as they obviously are not placing orders the same size as Nvidia or AMD.

I wonder what this portends for SambaNova and other similar vendors as well.

Re: Groq CEO: 'We No Longer Sell Hardware'

#39

Post by rattray »

ipsum2 wrote: post #19
gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

It's not really a lie though. They require 20x more chips (storing all the weights in sram instead of hbm is expensive!) than Nvidia GPUs for a ~2x speed increase. Overall the power cost is more expensive for groq than GPUs.

Source? Or how do you know that?

Re: Groq CEO: 'We No Longer Sell Hardware'

#40

Post by gandalfgeek »

halflings wrote: post #8
gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

No HBM because they use tons of fast SRAM instead. Isn't that the main driver for performance here? (the way I understood it => it's still cost effective at scale due to throughput increase this brings)

> No HBM because they use tons of fast SRAM instead. Isn't that the main driver for performance here?

No doubt fast SRAM helps, but from a computation pov imho its that they've statically planned computation and eliminated all locks.

Short explainer here: https://www.youtube.com/watch?v=H77tV1KcWIE (Based on their paper).

Post reply on HN