Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groq CEO: 'We No Longer Sell Hardware'

eetimes.com

Post reply on HN
Print view View original

141–150 of 152 posts

Re: Groq CEO: 'We No Longer Sell Hardware'

#141

Post by cyanydeez »

laserbeam wrote: post #75
fitzn wrote: post #53

Earlier quoted context omitted.

What open source model are you using when you hit groq? I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.

> 1.6 vs 1.8-2.2 seconds I believe certain companies would kill for 20% performance improvements on their main product.

What is the killer app product of a LLM Play ATM that's not a lossleader?

Re: Groq CEO: 'We No Longer Sell Hardware'

#142

Post by boroboro4 »

qeternity wrote: post #114
ben_w wrote: post #112

Earlier quoted context omitted.

When it's already faster than I can absorb the response, which for me as an organic brain includes the normal token generation rate of the free tier of ChatGPT. If I was using them to process far more text, e.g. summarise long documents, or if I was using it as an inline editing assistant, then I'd care more about the speed.

> When it's already faster than I can absorb the response Streaming a response from a chatbot is only one use-case of LLMs. I would argue the most interesting applications do not fall into this category.

Name one use case where there is a difference between latency of 200 t/s (fireworks.ai mixtral model) and 500 t/s (groq mixtral)? Not throughput and not time to first token, but latency.

Groq model shines at latency, not at the other two.

Re: Groq CEO: 'We No Longer Sell Hardware'

#143

Post by LoganDark »

SuchAnonMuchWow wrote: post #134
LoganDark wrote: post #24

That sucks. I wanted to save up for a couple years and get some hardware for home, but I guess the "AI" space moves so fast you barely get a couple months

You would need ~250 groq cards to run a 7B model since their system doesn't scale down. So if you want to buy their hardware, you need a few millions dollars. Their hardware was never for people at home, but for cloud providers.

That doesn't sound right. Their public demo ran on 568 LPUs because they had Mixtral-8x7B and LLaMA-70B (45B and 70B respectively). IIRC their cards each have slightly over 200MB of SRAM so this almost exactly checks out.

A 7B model would then be able to run on about 60 LPUs. Even at $20,000 per card that would be only $1.2 million and I highly doubt the cost is actually that high, that's just what DigiKey says the cost of an LPU is, if you're trying to buy just one :)

Re: Groq CEO: 'We No Longer Sell Hardware'

#144

Post by sebastiennight »

vinay_ys wrote: post #138

This business model is bound to get attacked and suffer a painful exit soon. Here's why: First, the whole systems of chips architecture that everyone is talking about will solve for increasing overall SRAM available to keep more model state on super fast memory and avoid going to slow memory. Secondly, anyone serious about their data (enterprises) won't be okay with making API calls to Groq. Anyone serious about thei…

The same has been said of OpenAI for a couple of years now (that they're just a platform to prototype on before moving on to open source models)...

... and yet, they're still leading the field.

I think it's a bit early to think the field is getting commoditized yet.

Re: Groq CEO: 'We No Longer Sell Hardware'

#145

Post by frozenport »

vinay_ys wrote: post #138

This business model is bound to get attacked and suffer a painful exit soon. Here's why: First, the whole systems of chips architecture that everyone is talking about will solve for increasing overall SRAM available to keep more model state on super fast memory and avoid going to slow memory. Secondly, anyone serious about their data (enterprises) won't be okay with making API calls to Groq. Anyone serious about thei…

>> won't be okay with making API calls to Groq

Linked article:

  If customers come with requests for high volumes of chips for very large installations, Groq will instead propose partnering on data center deployment

Re: Groq CEO: 'We No Longer Sell Hardware'

#146

Post by metadat »

vineyardmike wrote: post #111
metadat wrote: post #78

Earlier quoted context omitted.

"kill", .. why would anyone kill for a fraction of a second in this case? Informed folks know that LLM hosters aren't raking in the big bucks. They're selling dreams and aspirations, and those are what's driving the funding.

Google has used LMs in search for years (just not trendy LLMs), and search is famously optimized to the millisecond. Visa uses LMs to perform fraud detection every time someone makes a transaction, which is also quite latency sensitive. I'm guessing "informed folks" aren't so informed about the broader market. OpenAI and Anthropic's APIs are obviously not latency-driven. Same with comparable LLM API resellers like Az…

I wish things were so simple!

Reducing latency doesn't automatically translate to winning the market or even increased revenue. There are tons of other variables such as functionality, marketing, back-office sales deals and partnerships. Lots of times, users can't even tell which service is objectively better (even though you and I have the know how and tools to measure and better know reality).

Unfortunately the technical angle is only one piece of the puzzle.

Re: Groq CEO: 'We No Longer Sell Hardware'

#148

Post by razodactyl »

beepbooptheory wrote: post #133
michaelt wrote: post #130

Earlier quoted context omitted.

I understand there are games using LLMs for NPC dialog, yes [1] > If there is a future with this idea, its gotta be just shipping the LLM with game right? Depends how high you can let your GPU requirements get :) [1] https://www.youtube.com/watch?v=Kw51fkRiKZU

FWIW to confused others, trying to extract something from that video, it looks like this game [1] is using this stuff. Based solely on the reviews and the game play videos (while definitely acknowledging its technically in development status), it kinda looks like long term profitability is the least of their concerns here... EDIT: Watching the videos, I am more and more confused by why this is even desirable. The com…

Absolutely on the mark with this comment. LLMs aren't magical end-goal technology. We have a while to go it seems before they've settled into all the use-cases and we've established what does and doesn't work.

Re: Groq CEO: 'We No Longer Sell Hardware'

#149

Post by sdwr »

beepbooptheory wrote: post #124
michaelt wrote: post #116

Earlier quoted context omitted.

Depends on your application. For example, if you're a game company and you want to use LLMs so your players can converse with nonplayer characters in natural language, replacing a multiple-choice conversation tree - you'd want that to be low latency, and you'd want it to be cheap.

But are people really going to do this? The cost here seems prohibitive unless you're doing a subscription type game (and even then I'm not sure). And the kinds of games that benefit from open ended dialogue attract players who just want to pay an upfront cost and have an adventure. (All the sudden having nightmares of getting billed for the conversations I have in the single player game I happen to be enjoying...) I…

It would probably look like an InfiniteCraft-style model, where conversation possibilities are saved, and new dialogue is computed as needed.

Small, bounded conversations, with problematic lines trimmed over time, striking a balance between possibility and self-contradiction.

I could see it working really well in a Mass Effect-type game.

Re: Groq CEO: 'We No Longer Sell Hardware'

#150

Post by halflings »

germanjoey wrote: post #25
halflings wrote: post #8

Earlier quoted context omitted.

No HBM because they use tons of fast SRAM instead. Isn't that the main driver for performance here? (the way I understood it => it's still cost effective at scale due to throughput increase this brings)

cost effective in what sense? groq doesn't achieve high efficiency, only low latency. but that's not done in a cost-effective way. compare sambanova achieving the same performance with 8 chips instead of 568, and with higher precision.

The # of chips is not the most important metric.

Most important, even ignoring latency, is throughput (tokens) per $$$. And according to their own benchmark [1] (famous last words :)), they're quite cost efficient.

[1] https://www.semianalysis.com/p/groq-inference-tokenomics-spe...

Post reply on HN