Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq CEO: 'We No Longer Sell Hardware'

eetimes.com

Post reply on HN
Print view View original

91–100 of 152 posts

Re: Groq CEO: 'We No Longer Sell Hardware'

#91

Post by bee_rider »

fitzn wrote: post #53
geor9e wrote: post #44

I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…

What open source model are you using when you hit groq? I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.

What context did I miss that implies they are using an open source model?

Re: Groq CEO: 'We No Longer Sell Hardware'

#92

Post by latchkey »

elbear wrote: post #88
latchkey wrote: post #76

Earlier quoted context omitted.

Thank you. I've been on HN since 2009. The most successful people I've seen here, are the ones that are transparent, honest and ethical. I'm not trying to point fingers, I'm just focused on building a sustainable business and listening to my customers needs. The only way I can do that is by communicating with everyone around me as clearly and openly as I can. All our customers will know exactly where they stand, at a…

Just wanted to say that I like how you see things. Also, if you need help with infrastructure automation and are interested in using Nix for increased reproducibility, get in touch. I know you said you want to keep lean and already have a pipeline of potential candidates, but just in case.

Hi Lucian, thank you for stepping up to the plate. Recognized and appreciated. We aren't quite in the hiring phase today, still bootstrapping and focused on growing with customer demand, but I have absolutely added you to my list for the future. Cheers!

Re: Groq CEO: 'We No Longer Sell Hardware'

#93

Post by gpapilion »

latchkey wrote: post #14

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…

It’s basically their minimum cluster size for a reasonable model requires 8ish racks of compute.

Semi analysis did some cost estimates, and I did some but you’re likely paying somewhere in the 12 million dollar range for the equipment to serve a single query using llama-70b. Compare that to a couple of gpus, and it’s easy to see why they are struggling to sell hardware, they can’t scale down.

Since they didn’t use hbm, you need to stich enough cards together to get the memory to hold your model. It takes a lot of 256mb cards to get to 64gb, and there isn’t a good way to try the tech out since a single rack really can’t serve an LLM.

The cloud provider path sounds riskier since that’s two capital intensive businesses, chip design and production and running a cloud service provider.

Re: Groq CEO: 'We No Longer Sell Hardware'

#94

Post by latchkey »

gpapilion wrote: post #93
latchkey wrote: post #14

Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…

It’s basically their minimum cluster size for a reasonable model requires 8ish racks of compute. Semi analysis did some cost estimates, and I did some but you’re likely paying somewhere in the 12 million dollar range for the equipment to serve a single query using llama-70b. Compare that to a couple of gpus, and it’s easy to see why they are struggling to sell hardware, they can’t scale down. Since they didn’t use hb…

This is some fantastic insight. That would also inflate the opex (power/space/staffing) needs as well as people need to consider all the networking gear to hook this stuff together. 400G nic/switches/cables aren't cheap and in some cases, very hard to obtain in any sort of quantity.

It does seem like an odd move in that case. I liken this to a company like Bitmain. Why sell the miners when you could just run them yourselves? Well, fact is that they do both. But in this case, Groq is turning off the sales. Who knows, maybe it just ends up being a temporary thing until they can sort all of the pieces out.

Re: Groq CEO: 'We No Longer Sell Hardware'

#95

Post by elbear »

latchkey wrote: post #92
elbear wrote: post #88

Earlier quoted context omitted.

Just wanted to say that I like how you see things. Also, if you need help with infrastructure automation and are interested in using Nix for increased reproducibility, get in touch. I know you said you want to keep lean and already have a pipeline of potential candidates, but just in case.

Hi Lucian, thank you for stepping up to the plate. Recognized and appreciated. We aren't quite in the hiring phase today, still bootstrapping and focused on growing with customer demand, but I have absolutely added you to my list for the future. Cheers!

Sure. Thank you for answering. Good luck with what you're doing!

Re: Groq CEO: 'We No Longer Sell Hardware'

#96

Post by frozenport »

gpapilion wrote: post #86
laserbeam wrote: post #75

Earlier quoted context omitted.

> 1.6 vs 1.8-2.2 seconds I believe certain companies would kill for 20% performance improvements on their main product.

I have lots of questions about how important latency is since you may be replacing many minutes or hours of a person’s time with undoubtedly a quicker response by any measure. This seems like a knee jerk reaction assuming latency is as important as it’s been with advertising. I’m not convinced latency matters as much as groqs material tries to claim it does.

I guess its tool calling? When you chain the LLMs together?

Re: Groq CEO: 'We No Longer Sell Hardware'

#97

Post by w-ll »

gpapilion wrote: post #86
laserbeam wrote: post #75

Earlier quoted context omitted.

> 1.6 vs 1.8-2.2 seconds I believe certain companies would kill for 20% performance improvements on their main product.

I have lots of questions about how important latency is since you may be replacing many minutes or hours of a person’s time with undoubtedly a quicker response by any measure. This seems like a knee jerk reaction assuming latency is as important as it’s been with advertising. I’m not convinced latency matters as much as groqs material tries to claim it does.

When has latency ever not mattered?

Let alone 'chat' use cases, but holding a reponse up for N*1.2 longer than it could holds all sorts of other resources up/down stream.

Re: Groq CEO: 'We No Longer Sell Hardware'

#98

Post by latchkey »

ojn wrote: post #80
LoganDark wrote: post #64

Earlier quoted context omitted.

I'll look into it, though seeing "contact us" always makes me think they're not going to sell a single unit to a home user. (With that said, Groq probably wouldn't either. You can technically buy LPUs for 20k each, without an expectation of support, but it takes tens of them to run Mixtral.) Tenstorrent also looks incredibly Python-specific (as in, everything including their SMI seems mostly Python-based) which doesn…

Most of the low-level pieces are in Rust, the TUI is written in Python and most of the remaining pieces are getting lowered down to the Rust libraries over time. (It was all Python up until ~6 months ago) EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.

Even talking to people there, my experience is that they are super nice!

Re: Groq CEO: 'We No Longer Sell Hardware'

#99

Post by mlazos »

ein0p wrote: post #73
mlazos wrote: post #42

The smoke and mirrors around groq are finally clearing. Truth is that their system is insanely expensive to maintain. hundreds (> 500 iirc) of chips to get wild tokens/s but the power and maintenance expense is crazy high for that number of chips. TCO just isn’t worth it

You don't know that. For one thing, their silicon costs are going to be relatively cheap. It's an old reliable, 14nm process, and compared to even Google's TPU this is a relatively simple chip. For another they _could_ be putting all that silicon to a good use, and by all indications they are. Because there's far less local memory movement, and weights are distributed throughout the system, even this 14nm system coul…

Peak H100 power consumption is 700W. Average power consumption of the groq card (from their own website) is 240W. With 576 chips it just doesn’t look good. How much is that millisecond perf gain worth it to end users?

That said I think their arch is super interesting. I just think that demo was way too hype when the actual system is pretty impractical.

Re: Groq CEO: 'We No Longer Sell Hardware'

#100

Post by ipsum2 »

ipsum2 wrote: post #19
gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

It's not really a lie though. They require 20x more chips (storing all the weights in sram instead of hbm is expensive!) than Nvidia GPUs for a ~2x speed increase. Overall the power cost is more expensive for groq than GPUs.

Edit: 200x more chips, not 20x.
Post reply on HN