Earlier quoted context omitted.
The report I read said that latest TSMC is 17K per wafer. How much less it is for 14nm I don't know.
The masks are the expensive part, not the wafers.
They are both fabulously expensive.
Groq CEO: 'We No Longer Sell Hardware'
71–80 of 152 posts
Re: Groq CEO: 'We No Longer Sell Hardware'
#72Earlier quoted context omitted.
If you read on, Groq said they would only sell hardware to US companies and outside companies would get cloud services, not the LPUs. I think the US government told them to keep the LPUs in-house since they could be the secret sauce for scale.
I'm not questioning the deployment strategy, I'm wondering why Saudi Aramco wants to access so much compute power that is highly specialized(?) for generative AI workloads. Or is it more general than that?
Partly supply-chain security I imagine. If generative AI does indeed become the next big thing much nicer to have a giant pile of hardware physically in your country than buying a drip feed from a foreign company.
Re: Groq CEO: 'We No Longer Sell Hardware'
#73The smoke and mirrors around groq are finally clearing. Truth is that their system is insanely expensive to maintain. hundreds (> 500 iirc) of chips to get wild tokens/s but the power and maintenance expense is crazy high for that number of chips. TCO just isn’t worth it
You don't know that. For one thing, their silicon costs are going to be relatively cheap. It's an old reliable, 14nm process, and compared to even Google's TPU this is a relatively simple chip. For another they _could_ be putting all that silicon to a good use, and by all indications they are. Because there's far less local memory movement, and weights are distributed throughout the system, even this 14nm system could be energy efficient. 9/10ths of all power in a conventional system does not go towards compute - it's wasted in moving data back and forth. This is especially bad in transformers, which, because of their size, largely defeat the memory hierarchies the architects worked so hard to perfect. IOW, all your caches are useless and you're unnecessarily wasting 90% of your energy while also getting worse latency and worse throughput (due to memory bus bandwidth constraints). Oops. These folks seem to be offering something that nobody else does - a feasible, proven way to get out of jail free. I wish them all the success they can get, because all the other currently available architectures are largely unsuitable for high throughput transformer inference, and they work in spite, instead of because, of their design.
Re: Groq CEO: 'We No Longer Sell Hardware'
#74Earlier quoted context omitted.
How do y'all compare to https://tensorwave.com ?
To be totally honest, I have no idea. When I first learned about them, I reached out to the CEO privately on LinkedIn, he asked what I was up to, I told him probably more than I should have (I come from an open source and transparent background), then he stopped talking to me entirely. Since then, one of the co-founders blocked me on Twitter for pointing out that despite their claims, they were not the first MI300x t…
For what it’s worth, to me, your approach is the one I’d prefer as a customer.
Re: Groq CEO: 'We No Longer Sell Hardware'
#75I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…
What open source model are you using when you hit groq? I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.
> 1.6 vs 1.8-2.2 seconds
I believe certain companies would kill for 20% performance improvements on their main product.
Re: Groq CEO: 'We No Longer Sell Hardware'
#76Earlier quoted context omitted.
To be totally honest, I have no idea. When I first learned about them, I reached out to the CEO privately on LinkedIn, he asked what I was up to, I told him probably more than I should have (I come from an open source and transparent background), then he stopped talking to me entirely. Since then, one of the co-founders blocked me on Twitter for pointing out that despite their claims, they were not the first MI300x t…
For what it’s worth, to me, your approach is the one I’d prefer as a customer.
Thank you. I've been on HN since 2009. The most successful people I've seen here, are the ones that are transparent, honest and ethical.
I'm not trying to point fingers, I'm just focused on building a sustainable business and listening to my customers needs. The only way I can do that is by communicating with everyone around me as clearly and openly as I can. All our customers will know exactly where they stand, at all times.
I post a lot of open information on r/AMD_Stock and the feedback that I've gotten there has been exceptional. People are excited to see if AMD can claw back a bit of the market. For the safety and success of AI, we don't need team blue vs. team red, we need everyone to work towards having as many options as possible.
This is one way that I think we are going to differentiate ourselves. We won't just have MI300x, we will have every best-of-the-best chunk of hardware that we can get our hands on. No longer will super computers be tied up behind govt/edu grants. We want to democratize it. It has long been a goal of mine to build a super computer, and here is my chance. I'm excited.
One thing that sets us apart is that my co-founder and I have a ton of experience deploying, managing and optimizing 150,000 AMD GPUs and 20PB+ of storage. We did it ourselves, all through covid and all of the supply chain issues. I'm not sure many others have done that and this is something that we are well versed at doing.
I'm also seeing my competitors hiring a ton, while we are staying lean and mean with a very small team. I'd rather automate everything we deploy and focus all of our investors money on buying compute. We also have a pool of previous people we can hire from, which I think is quite an advantage over blanket hiring.
Re: Groq CEO: 'We No Longer Sell Hardware'
#77I don't understand why the comments are trash-talking Groq. They are the fastest LLM inference provider by a big margin. Why would they sell their hardware to any other company for any price? Keep it all for themselves and take over the market. 95% of my LLM requests go to Groq these days because it's 0.25 seconds round trip for a complete answer. In comparison, "Claude Instant" takes about 4 seconds. The other 5% of…
What open source model are you using when you hit groq? I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.
[deleted]
Re: Groq CEO: 'We No Longer Sell Hardware'
#78Earlier quoted context omitted.
What open source model are you using when you hit groq? I just benchmarked some perf for some of my larger context window queries last week and groq's API took 1.6 seconds versus 1.8 to 2.2 for OpenAI GPT-3.5-turbo. So, it wasn't much faster. I almost emailed their support to see if I was doing something wrong. Would love to hear any details about your workload or the complexity of your queries.
> 1.6 vs 1.8-2.2 seconds I believe certain companies would kill for 20% performance improvements on their main product.
"kill", .. why would anyone kill for a fraction of a second in this case? Informed folks know that LLM hosters aren't raking in the big bucks.
They're selling dreams and aspirations, and those are what's driving the funding.
Re: Groq CEO: 'We No Longer Sell Hardware'
#79Interesting, I guess that is why I never got a response back from them about buying their stuff. My guess is that they realized that just selling hardware is a lot harder than running it themselves. Deploying this level of compute is non-trivial, with very high rates of failure, as well as huge supply chain issues. If you have to sell the hardware and support people buying it, that is a world of trouble. > no-one wan…
How's the overall software support for MI300 series? The hardware itself looks great.
(also, +100 to valuing honesty and transparency)
Re: Groq CEO: 'We No Longer Sell Hardware'
#80Earlier quoted context omitted.
Save up for Tenstorrent instead.
I'll look into it, though seeing "contact us" always makes me think they're not going to sell a single unit to a home user. (With that said, Groq probably wouldn't either. You can technically buy LPUs for 20k each, without an expectation of support, but it takes tens of them to run Mixtral.) Tenstorrent also looks incredibly Python-specific (as in, everything including their SMI seems mostly Python-based) which doesn…
Most of the low-level pieces are in Rust, the TUI is written in Python and most of the remaining pieces are getting lowered down to the Rust libraries over time.
(It was all Python up until ~6 months ago)
EDIT: Oh, and you can buy the Grayskull cards online now, without contacting anyone.