Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq CEO: 'We No Longer Sell Hardware'

eetimes.com

Post reply on HN
Print view View original

121–130 of 152 posts

Re: Groq CEO: 'We No Longer Sell Hardware'

#121

Post by verdverm »

robrenaud wrote: post #90
laserbeam wrote: post #75

Earlier quoted context omitted.

> 1.6 vs 1.8-2.2 seconds I believe certain companies would kill for 20% performance improvements on their main product.

Model quality matters a ton too. They aren't serving OpenAI or Anthropic models, which are state of the art.

Research suggest most answers and use cases do not require the largest, most sophisticated models. When you start building more complex systems, the overall time increases from chaining and you can pick different models for different points

Re: Groq CEO: 'We No Longer Sell Hardware'

#122

Post by thawab »

mike_hearn wrote: post #115
thawab wrote: post #43

Earlier quoted context omitted.

It’s about material discovery[0], Aramco built a 250b model to help them improve efficiency [1]. [0] https://deepmind.google/discover/blog/millions-of-new-materi... [1] https://www.aramco.com/en/news-media/speeches/2024/leap24--r...

What's the connection there? The second link says it's not about materials discovery at all but rather their GigaPOWERS model, which is a physics simulation used to optimize CO2 injection into their fields (i.e. optimizing recovery). POWERS is old, it was in development for decades already. Given that they don't plan to use Groq for LLMs but simply for its parallel computation abilities. I wonder to what extent this…

It's an assumption from my side, that they will utilize GenAI in material discovery for Aramco or SABIC[0]. Even if Groq won't fit that use-case a couple of billions with another hardware vendor is nothing if this paid off.

[0]https://en.wikipedia.org/wiki/SABIC

Re: Groq CEO: 'We No Longer Sell Hardware'

#123

Post by mlazos »

ein0p wrote: post #108
mlazos wrote: post #99

Earlier quoted context omitted.

Peak H100 power consumption is 700W. Average power consumption of the groq card (from their own website) is 240W. With 576 chips it just doesn’t look good. How much is that millisecond perf gain worth it to end users? That said I think their arch is super interesting. I just think that demo was way too hype when the actual system is pretty impractical.

So? They aren't performing the same computation. You can't compare the two. What you can compare is power draw at an equivalent tokens/sec on the same model for the entire system. But you don't have that number.

I’m just estimating here from public numbers. My point is the power consumption could be 100W on that workload and the groq chip could be 1k, both ridiculously optimistic. The whole system is still crazy expensive. H100s will not have latency as fast, but in terms of concurrent users and TCO, I really don’t think groq will be worth it. You could probably get the same concurrent users and throughput with like 8 h100s. Latency won’t be as good, but price could be much lower.

Re: Groq CEO: 'We No Longer Sell Hardware'

#124

Post by beepbooptheory »

michaelt wrote: post #116
gpapilion wrote: post #86

Earlier quoted context omitted.

I have lots of questions about how important latency is since you may be replacing many minutes or hours of a person’s time with undoubtedly a quicker response by any measure. This seems like a knee jerk reaction assuming latency is as important as it’s been with advertising. I’m not convinced latency matters as much as groqs material tries to claim it does.

Depends on your application. For example, if you're a game company and you want to use LLMs so your players can converse with nonplayer characters in natural language, replacing a multiple-choice conversation tree - you'd want that to be low latency, and you'd want it to be cheap.

But are people really going to do this? The cost here seems prohibitive unless you're doing a subscription type game (and even then I'm not sure). And the kinds of games that benefit from open ended dialogue attract players who just want to pay an upfront cost and have an adventure.

(All the sudden having nightmares of getting billed for the conversations I have in the single player game I happen to be enjoying...)

If there is a future with this idea, its gotta be just shipping the LLM with game right?

Re: Groq CEO: 'We No Longer Sell Hardware'

#126

Post by Const-me »

beepbooptheory wrote: post #124
michaelt wrote: post #116

Earlier quoted context omitted.

Depends on your application. For example, if you're a game company and you want to use LLMs so your players can converse with nonplayer characters in natural language, replacing a multiple-choice conversation tree - you'd want that to be low latency, and you'd want it to be cheap.

But are people really going to do this? The cost here seems prohibitive unless you're doing a subscription type game (and even then I'm not sure). And the kinds of games that benefit from open ended dialogue attract players who just want to pay an upfront cost and have an adventure. (All the sudden having nightmares of getting billed for the conversations I have in the single player game I happen to be enjoying...) I…

> If there is a future with this idea, its gotta be just shipping the LLM with game right?

That might be a nice application for this library of mine: https://github.com/Const-me/Cgml/

That’s an open source Mistral ML model implementation which runs on GPUs (all of them, not just nVidia), takes 4.5GB on disk, uses under 6GB of VRAM, and optimized for interactive single-user use case. Probably fast enough for that application.

You wouldn’t want in-game dialogues with the original model though. Game developers would need to finetune, retrain and/or do something else with these weights and/or my implementation.

Re: Groq CEO: 'We No Longer Sell Hardware'

#127

Post by jkachmar »

alted wrote: post #35

Custom state-of-the-art silicon is ridiculously expensive. For a minimum 100 wafers = 10k chips, Groq may have paid $100M = $10k/chip purely in amortizing design costs. Chip design (software + engineer time) and fabrication setup (lithography masks) grow exponentially [1][2] with smaller nodes, e.g., maybe $100M for Groq's current 14nm chips to ~$500M for their planned 4nm tapeout. Once you reach mass production (>>1…

i don’t support hardware development directly, but i’m a software infrastructure engineer working adjacent to the teams that do so.

can’t comment on specifics, but imo our hardware team punches above its weight class in terms of # of people and time spent in design.

Re: Groq CEO: 'We No Longer Sell Hardware'

#128

Post by YetAnotherNick »

fitzn wrote: post #53
geor9e wrote: post #44

I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…

What open source model are you using when you hit groq? I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.

It's not a lot more faster for input but it is something like 10x faster for output(mixtral vs gpt-3.5). This could enable completely new mode of interaction with LLMs e.g. agents.

In most of the cases, overall response time is mostly dominated by output as it is ~100x slower per token than input.

Re: Groq CEO: 'We No Longer Sell Hardware'

#129

Post by freediver »

geor9e wrote: post #44

I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…

How do you decide which requests to send to gpt4/opus?

Re: Groq CEO: 'We No Longer Sell Hardware'

#130

Post by michaelt »

beepbooptheory wrote: post #124
michaelt wrote: post #116

Earlier quoted context omitted.

Depends on your application. For example, if you're a game company and you want to use LLMs so your players can converse with nonplayer characters in natural language, replacing a multiple-choice conversation tree - you'd want that to be low latency, and you'd want it to be cheap.

But are people really going to do this? The cost here seems prohibitive unless you're doing a subscription type game (and even then I'm not sure). And the kinds of games that benefit from open ended dialogue attract players who just want to pay an upfront cost and have an adventure. (All the sudden having nightmares of getting billed for the conversations I have in the single player game I happen to be enjoying...) I…

I understand there are games using LLMs for NPC dialog, yes [1]

> If there is a future with this idea, its gotta be just shipping the LLM with game right?

Depends how high you can let your GPU requirements get :)

[1] https://www.youtube.com/watch?v=Kw51fkRiKZU

Post reply on HN