Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq CEO: 'We No Longer Sell Hardware'

eetimes.com

Post reply on HN
Print view View original

131–140 of 152 posts

Re: Groq CEO: 'We No Longer Sell Hardware'

#131

Post by lostmsu »

latchkey wrote: post #63
htrp wrote: post #62

Earlier quoted context omitted.

Shiny! You have you guys gotten MI300X's working for non-inference use cases?

We got them, then gave them to a customer to use for a week, got them back and now we are having some hardware issues that we are in the process of sorting out. I literally haven't had more than a few hours time on them yet. They _should_ work fine for both training and inference, but since nobody has done much in the way of public in-depth benchmarks yet... I was hoping to get people to do it for us in order to stay…

That sounds like your machine broke after 1 week of intensive use.

Re: Groq CEO: 'We No Longer Sell Hardware'

#132

Post by c-c-c-c-c »

latchkey wrote: post #14

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…

The GroqCard (RS-GQ-GC1-0109) was in stock at mouser a few weeks ago and they are still taking orders.

Re: Groq CEO: 'We No Longer Sell Hardware'

#133

Post by beepbooptheory »

michaelt wrote: post #130
beepbooptheory wrote: post #124

Earlier quoted context omitted.

But are people really going to do this? The cost here seems prohibitive unless you're doing a subscription type game (and even then I'm not sure). And the kinds of games that benefit from open ended dialogue attract players who just want to pay an upfront cost and have an adventure. (All the sudden having nightmares of getting billed for the conversations I have in the single player game I happen to be enjoying...) I…

I understand there are games using LLMs for NPC dialog, yes [1] > If there is a future with this idea, its gotta be just shipping the LLM with game right? Depends how high you can let your GPU requirements get :) [1] https://www.youtube.com/watch?v=Kw51fkRiKZU

FWIW to confused others, trying to extract something from that video, it looks like this game [1] is using this stuff. Based solely on the reviews and the game play videos (while definitely acknowledging its technically in development status), it kinda looks like long term profitability is the least of their concerns here...

EDIT: Watching the videos, I am more and more confused by why this is even desirable. The complexity of dialogue in a game, it seems, needs to match the complexity of the more general possibilities and actions you can undertake in the game itself. Without that, it all just feels like you are in a little chatbot sandbox within the game, even if the dialogue is perfectly "in character." It all seems to feel less immersive with the LLMs.

1. https://store.steampowered.com/app/2240920/Vaudeville/

Re: Groq CEO: 'We No Longer Sell Hardware'

#134

Post by SuchAnonMuchWow »

LoganDark wrote: post #24

That sucks. I wanted to save up for a couple years and get some hardware for home, but I guess the "AI" space moves so fast you barely get a couple months

You would need ~250 groq cards to run a 7B model since their system doesn't scale down. So if you want to buy their hardware, you need a few millions dollars.

Their hardware was never for people at home, but for cloud providers.

Re: Groq CEO: 'We No Longer Sell Hardware'

#135

Post by latchkey »

winwang wrote: post #103
latchkey wrote: post #85

Earlier quoted context omitted.

We haven't spec'd to buy A's quite yet as you're actually the first person I've heard even suggest them. If you're truly interested, hit me up personally. By default, we are putting dual 9754's in the chassis, along with 3TB ram and 155TB nvme. A pretty beefy box. However, if you want to work with us, we can customize this to whatever customers need. Effectively, we are the capex/opex for something that requires a lo…

Was mostly just checking to see if someone had already tested GPGPU, though I know some HPC labs like the MI300A. While I am starting a business, I'm not at the point of shipping software just yet (I wish!). Will definitely keep you in mind for if/when we get to AMD -- it's something I'd want, though that depends on achieving any modicum of success, haha.

Awesome, thanks for the follow up. I wish you the best with your business! It definitely isn't easy, but we will be here to help, when you need it.

Re: Groq CEO: 'We No Longer Sell Hardware'

#136

Post by latchkey »

lostmsu wrote: post #131
latchkey wrote: post #63

Earlier quoted context omitted.

We got them, then gave them to a customer to use for a week, got them back and now we are having some hardware issues that we are in the process of sorting out. I literally haven't had more than a few hours time on them yet. They _should_ work fine for both training and inference, but since nobody has done much in the way of public in-depth benchmarks yet... I was hoping to get people to do it for us in order to stay…

That sounds like your machine broke after 1 week of intensive use.

Sadly, not even intensive.

The thing that nobody talks about is that there is a high rate of failures on this high end equipment. I've heard as high as 20%, in the first month. I'm not even talking about AMD here.

If anyone thinks they can just buy some accelerators and throw them into a rack and expect them to work flawlessly... they've got some hard lessons to learn.

This will be less of an issue as we grow as we will have plenty of stock to pull from, but it is a real bummer as we are starting as a proof of concept first. We started working on this business last August, before anyone knew whether or not AMD would even change course on AI.

The good news is that we onboarded a customer the day that we announced our availability, we passed that PoC challenge with flying colors and closed significant additional funding immediately after that. Onwards and upwards, just have to roll with the punches.

Re: Groq CEO: 'We No Longer Sell Hardware'

#137

Post by touisteur »

latchkey wrote: post #136
lostmsu wrote: post #131

Earlier quoted context omitted.

That sounds like your machine broke after 1 week of intensive use.

Sadly, not even intensive. The thing that nobody talks about is that there is a high rate of failures on this high end equipment. I've heard as high as 20%, in the first month. I'm not even talking about AMD here. If anyone thinks they can just buy some accelerators and throw them into a rack and expect them to work flawlessly... they've got some hard lessons to learn. This will be less of an issue as we grow as we w…

Ooooh yeah, from experience, even the usual (thorough) suspects who over-over-over-spec cooling and power (as anyone here has heard an HPE Apollo 6500 at full fan speed can attest) have a hard time getting all the interconnects, pcie, firmware stuff up to snuff and running H24. Once it's setup it's amazing but the bringup can be rocky.

And I'm not even talking 100/400G network, wonderful wonderful hardware, good luck debugging and getting all the RoCE/RDMA/GPUDirect/StorageDirect/NCCL working (already a bit of pain on nvidia, with a large installed base...).

Either you want to learn all this stuff (for reasons) or you're dumping a lot of money on fast-evolving tech.

Re: Groq CEO: 'We No Longer Sell Hardware'

#138

Post by vinay_ys »

This business model is bound to get attacked and suffer a painful exit soon. Here's why:

First, the whole systems of chips architecture that everyone is talking about will solve for increasing overall SRAM available to keep more model state on super fast memory and avoid going to slow memory.

Secondly, anyone serious about their data (enterprises) won't be okay with making API calls to Groq. Anyone serious about their data and have a lot scale (consumer internet) won't also be okay with making expensive API calls to Groq at scale.

Their cloud is attractive only if I can use their API for experimentation toy apps to continue developing in this direction while the rest of the major industry players systems of chip architecture catches up and solves for SRAM size bottleneck and manufacturing process bottleneck, and once that's solved, I get more powerful compute for cheaper $$ to deploy on-prem.

So, this cloud strategy is short-lived. I see another pivot on the horizon.

Re: Groq CEO: 'We No Longer Sell Hardware'

#139

Post by zachbee »

Totally saw this one coming! [1]

I think one major challenge they'll face is that their architecture is incredibly fast at running the ~10-100B parameter open-source models, but starts hitting scaling issues with state-of-the-art models. They need 10k+ chips for a GPT-4-class model, but their optical interconnect only supports a few hundred chips.

[1] https://www.zach.be/p/why-is-everybody-talking-about-groq

Re: Groq CEO: 'We No Longer Sell Hardware'

#140

Post by zachbee »

geor9e wrote: post #44

I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…

If I was developing an AI app, I'd care about quality first before speed. And the open-source models just aren't as good as the closed ones.
Post reply on HN