Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq CEO: 'We No Longer Sell Hardware'

eetimes.com

Post reply on HN
Print view View original

51–60 of 152 posts

Re: Groq CEO: 'We No Longer Sell Hardware'

#51

Post by ethbr1 »

ipsum2 wrote: post #19
gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

It's not really a lie though. They require 20x more chips (storing all the weights in sram instead of hbm is expensive!) than Nvidia GPUs for a ~2x speed increase. Overall the power cost is more expensive for groq than GPUs.

Are power and 20x 14-nm chip capacity limiting factors currently?

It's not inconceivable that's a better trade-off than leading-node and HBM requirements.

Re: Groq CEO: 'We No Longer Sell Hardware'

#52

Post by huac »

LoganDark wrote: post #28
dsrtslnd23 wrote: post #9

So unless there are new Croq datacenters coming, this is only interesting for North American users. Otherwise H100 based latency optimized solutions would be faster - in particular for time-to-first-token sensitive applications.

> latency optimized solutions would be faster - in particular for time-to-first-token sensitive applications Do you have any idea how fast Groq is? Go try it. Consistently over 400 t/s for most of the models that they support, and extremely low latency.

time to first token != tokens per second

remember that EU -> US is ~150ms unavoidable latency, for example. then your comparison is local H100 vs Grok + 150ms latency to first token.

Re: Groq CEO: 'We No Longer Sell Hardware'

#53

Post by fitzn »

geor9e wrote: post #44

I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…

What open source model are you using when you hit groq?

I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.

Re: Groq CEO: 'We No Longer Sell Hardware'

#54

Post by karma_pharmer »

shrubble wrote: post #47
alted wrote: post #35

Custom state-of-the-art silicon is ridiculously expensive. For a minimum 100 wafers = 10k chips, Groq may have paid $100M = $10k/chip purely in amortizing design costs. Chip design (software + engineer time) and fabrication setup (lithography masks) grow exponentially [1][2] with smaller nodes, e.g., maybe $100M for Groq's current 14nm chips to ~$500M for their planned 4nm tapeout. Once you reach mass production (>>1…

The report I read said that latest TSMC is 17K per wafer. How much less it is for 14nm I don't know.

The masks are the expensive part, not the wafers.

Re: Groq CEO: 'We No Longer Sell Hardware'

#55

Post by ethbr1 »

dralley wrote: post #10
gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

So Itanium and its "sufficiently smart compiler" but functional?

From skimming the link above, it seems like they accepted it's extremely difficult (maybe impossible) to generate high ILP from a VLIW compiler on complex hardware (what Itanium tried to do).

So they attacked the italicized portion and simplified the hardware. Mostly by eliminating memory-layer non-determinism / using time-sync'd global memory instructions as part of the ISA(?).

This apparently reduced the difficulty of the compiler problem to something manageable (but no doubt still "fun")... and voila, performance.

Re: Groq CEO: 'We No Longer Sell Hardware'

#57

Post by ethbr1 »

aleph_minus_one wrote: post #27
latchkey wrote: post #14

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…

> If you have to sell the hardware and support people buying it, that is a world of trouble. What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription?

> What is the difference between this and having to sell the cloud access and supporting the people who buy a subscription

Knowledge/training.

If you're shipping a brand new hardware arch, exposed as raw hardware, then you're on the hook for training everyone in the world and fixing all their weird edge case uses.

I.e. are you willing to invest in Intel/AMD/Nvidia-scale QA and support?

If you're exposing a PaaS (or even IaaS), then you have some levers you can tweak / mask behind the scenes, so only your team need be experts at low-level operations.

For a fast-paced company, the latter model makes a lot more sense, at least until hardware+software stabilizes.

Re: Groq CEO: 'We No Longer Sell Hardware'

#58

Post by brookst »

aleph_minus_one wrote: post #31
creato wrote: post #22

Earlier quoted context omitted.

I'm not questioning the deployment strategy, I'm wondering why Saudi Aramco wants to access so much compute power that is highly specialized(?) for generative AI workloads. Or is it more general than that?

> I'm wondering why Saudi Aramco wants to access so much compute power that is highly specialized(?) for generative AI workloads. Or is it more general than that? For vanity reasons and because AI is the future (not every company acts that rationally for huge buying decisions).

Also diversification. They’re smart people; oil isn’t the future. Comparatively small investments to hedge make perfect sense.

Re: Groq CEO: 'We No Longer Sell Hardware'

#59

Post by LoganDark »

huac wrote: post #52
LoganDark wrote: post #28

Earlier quoted context omitted.

> latency optimized solutions would be faster - in particular for time-to-first-token sensitive applications Do you have any idea how fast Groq is? Go try it. Consistently over 400 t/s for most of the models that they support, and extremely low latency.

time to first token != tokens per second remember that EU -> US is ~150ms unavoidable latency, for example. then your comparison is local H100 vs Grok + 150ms latency to first token.

> time to first token != tokens per second

I said "and extremely low latency" because I know they are different. Groq's TTFT is still consistently competitive with any other provider, and lower than most of them. Here's some benchmarks: https://github.com/ray-project/llmperf-leaderboard#70b-model...

Re: Groq CEO: 'We No Longer Sell Hardware'

#60

Post by anon291 »

ethbr1 wrote: post #55
dralley wrote: post #10

Earlier quoted context omitted.

So Itanium and its "sufficiently smart compiler" but functional?

From skimming the link above, it seems like they accepted it's extremely difficult (maybe impossible) to generate high ILP from a VLIW compiler on complex hardware (what Itanium tried to do). So they attacked the italicized portion and simplified the hardware. Mostly by eliminating memory-layer non-determinism / using time-sync'd global memory instructions as part of the ISA(?). This apparently reduced the difficulty…

The problem set groq is restricted to (known size tensor manipulation) lends itself to much easier solutions than full blown ILP. The general problem of compiling arbitrary Turing complete algorithms to the arch is NP hard. Tensor manipulation... That's a different story.
Post reply on HN