Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq CEO: 'We No Longer Sell Hardware'

eetimes.com

Post reply on HN
Print view View original

101–110 of 152 posts

Re: Groq CEO: 'We No Longer Sell Hardware'

#101

Post by LoganDark »

ojn wrote: post #80
LoganDark wrote: post #64

Earlier quoted context omitted.

I'll look into it, though seeing "contact us" always makes me think they're not going to sell a single unit to a home user. (With that said, Groq probably wouldn't either. You can technically buy LPUs for 20k each, without an expectation of support, but it takes tens of them to run Mixtral.) Tenstorrent also looks incredibly Python-specific (as in, everything including their SMI seems mostly Python-based) which doesn…

Most of the low-level pieces are in Rust, the TUI is written in Python and most of the remaining pieces are getting lowered down to the Rust libraries over time. (It was all Python up until ~6 months ago) EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.

> EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.

I actually don't mind having to contact, I only mind if they won't want to sell to me due to being a non-bulk order.

> Most of the low-level pieces are in Rust

That's awesome!

Re: Groq CEO: 'We No Longer Sell Hardware'

#102

Post by torginus »

gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

I wonder if the use of eDRAM (https://en.wikipedia.org/wiki/EDRAM), which is essentially embedding DRAM into a chip made on a logic process would be a good idea here.

EDRAM is essentially a tradeoff between SRAM and DRAM, offering much greater density at the cost of somewhat worse throughput and latency.

There were a couple of POWER cpus that used EDRAM as L3 cache, but it seems to have fallen out of favor.

Re: Groq CEO: 'We No Longer Sell Hardware'

#103

Post by winwang »

latchkey wrote: post #85
winwang wrote: post #83

Earlier quoted context omitted.

Nice! I'm pretty interested in GPGPU applications and MI300A, but I'm also just glad for more competition. Love that you hit up the LocalLLaMa sub. Do you know if anyone's tested CuPy stuff on MI300X?

We haven't spec'd to buy A's quite yet as you're actually the first person I've heard even suggest them. If you're truly interested, hit me up personally. By default, we are putting dual 9754's in the chassis, along with 3TB ram and 155TB nvme. A pretty beefy box. However, if you want to work with us, we can customize this to whatever customers need. Effectively, we are the capex/opex for something that requires a lo…

Was mostly just checking to see if someone had already tested GPGPU, though I know some HPC labs like the MI300A. While I am starting a business, I'm not at the point of shipping software just yet (I wish!). Will definitely keep you in mind for if/when we get to AMD -- it's something I'd want, though that depends on achieving any modicum of success, haha.

Re: Groq CEO: 'We No Longer Sell Hardware'

#104

Post by Oribi »

mlazos wrote: post #42

The smoke and mirrors around groq are finally clearing. Truth is that their system is insanely expensive to maintain. hundreds (> 500 iirc) of chips to get wild tokens/s but the power and maintenance expense is crazy high for that number of chips. TCO just isn’t worth it

Why would they want to run it themselves if the TCO didn’t work out

Re: Groq CEO: 'We No Longer Sell Hardware'

#105

Post by mrkeen »

Oribi wrote: post #104
mlazos wrote: post #42

The smoke and mirrors around groq are finally clearing. Truth is that their system is insanely expensive to maintain. hundreds (> 500 iirc) of chips to get wild tokens/s but the power and maintenance expense is crazy high for that number of chips. TCO just isn’t worth it

Why would they want to run it themselves if the TCO didn’t work out

I thought that was par for the course these days.

Operate at a loss. Get a big valuation. Cash out.

Re: Groq CEO: 'We No Longer Sell Hardware'

#106

Post by tpetry »

huac wrote: post #48
geor9e wrote: post #44

I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…

why don't you stream the results?

You still have to wait for the end of the streamed response until you can continue with your task.

Re: Groq CEO: 'We No Longer Sell Hardware'

#107

Post by grandmczeb »

torginus wrote: post #102
gandalfgeek wrote: post #6

They're calling the lie on needing bleeding edge hardware for performance. 5 yr old silicon (14 nm!!) and no hbm. Their secret sauce seems to be an ahead-of-time compiler that statically lays out entire computation, enabling zero contention at runtime. Basically, they stamp out all non-determinism. https://wow.groq.com/isca-2022-paper

I wonder if the use of eDRAM ( https://en.wikipedia.org/wiki/EDRAM ), which is essentially embedding DRAM into a chip made on a logic process would be a good idea here. EDRAM is essentially a tradeoff between SRAM and DRAM, offering much greater density at the cost of somewhat worse throughput and latency. There were a couple of POWER cpus that used EDRAM as L3 cache, but it seems to have fallen out of favor.

It fell out of favor because it lost the density advantage in newer processes.

Re: Groq CEO: 'We No Longer Sell Hardware'

#108

Post by ein0p »

mlazos wrote: post #99
ein0p wrote: post #73

Earlier quoted context omitted.

You don't know that. For one thing, their silicon costs are going to be relatively cheap. It's an old reliable, 14nm process, and compared to even Google's TPU this is a relatively simple chip. For another they _could_ be putting all that silicon to a good use, and by all indications they are. Because there's far less local memory movement, and weights are distributed throughout the system, even this 14nm system coul…

Peak H100 power consumption is 700W. Average power consumption of the groq card (from their own website) is 240W. With 576 chips it just doesn’t look good. How much is that millisecond perf gain worth it to end users? That said I think their arch is super interesting. I just think that demo was way too hype when the actual system is pretty impractical.

So? They aren't performing the same computation. You can't compare the two. What you can compare is power draw at an equivalent tokens/sec on the same model for the entire system. But you don't have that number.

Re: Groq CEO: 'We No Longer Sell Hardware'

#109

Post by Havoc »

htrp wrote: post #23
Havoc wrote: post #20

Given that their hardware is different I can kinda see how they don’t want to deal with supporting customers. > what do you mean I can’t just drop a CUDA docker image in?

if you're a hardware startup that doesn't sell hardware, what are you?

Hardware setup that produces superior hardware and extracts the benefit in house ?

Re: Groq CEO: 'We No Longer Sell Hardware'

#110

Post by b-side »

Oribi wrote: post #104
mlazos wrote: post #42

The smoke and mirrors around groq are finally clearing. Truth is that their system is insanely expensive to maintain. hundreds (> 500 iirc) of chips to get wild tokens/s but the power and maintenance expense is crazy high for that number of chips. TCO just isn’t worth it

Why would they want to run it themselves if the TCO didn’t work out

Because they rather operate at a loss with high revenue rather than have 0 revenue and loss?
Post reply on HN