Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Growing Compute Shortage

apollo.com

Post reply on HN
Print view View original

61–70 of 99 posts

Re: The Growing Compute Shortage

#61

Post by senordevnyc »

cmiles8 wrote: post #48

Four things are currently correct: 1. There is a huge demand for compute, specifically GPU compute 2. Infrastructure providers are building like crazy, including taking on massive debt to fund this because their own cash flow can’t cover the bills 3. The demand for that compute is broadly being paid for with investor dollars pumping up the valuation of AI companies, not cash flow from said companies. If those subsidi…

Eh, I don’t think there’s any reason to think #3 is true, and the whole thing being a house of cards is predicated on that one.

If the massive demand is still present for compute at market rates (which I believe it is), then your second point is just investors spending cap ex to build out valuable and profitable assets, no problem there.

Time will tell.

Re: The Growing Compute Shortage

#62

Post by mutkach »

reticulates wrote: post #13

Yes but how much of that compute shortage is from demand that is subsidized? We’ve seen companies like Uber drastically cut how much they are willing to spend on AI because they are paying actual usage costs, while at the same time OpenAI and Anthropic increase the limits on their fixed cost plans for individuals meaning people not paying usage costs are using it more and more… doesn’t this show that the compute shor…

True, and even selling token price may not be illustrative of what it actually costs them to provide the service. Tokens may be sold at a loss if majority of spenders are running agents 24/7

The industry and users moves on from single chat-based to more and more "agentic" workflows that may generate longer workloads with multiple simultaneous agents (separate agents - separate contexts - separate KV caches).

My estimation is based on, say, running a Kimi3 on a 24 B200 GPUs - it is very easy to lose money when selling tokens at "market" prices.

Re: The Growing Compute Shortage

#63

Post by reticulates »

tristanj wrote: post #53
reticulates wrote: post #41

Earlier quoted context omitted.

Where are you getting that 80% figure from? Even semi analysis, the most aggressively optimistic analysts, put it at around 60%. https://newsletter.semianalysis.com/p/anthropic-growth-and-b...

It's clearly listed in the article I linked. The number comes from SemiAnalysis, from a newer report than the one you cited.

Looks like it’s behind a paywall. I’ll take their word for it that semi analysis now estimate it to be 80%. That makes my point even stronger, if that number is true, where is the money? The report says that Anthropic generate over $50bn in revenue so at 80% margins that gives $40bn in profit. Where is that money? If they’re generating $40bn in profit, even after accounting for very high employee compensation and training costs… they should have tens of billions in profit, yet they’re out raising tens of billions instead. Where is the money going? And if only 20% is their actual inference costs, where are all these compute providers going to make their money? The world is at compute capacity on, what, $10bn in revenue?

Re: The Growing Compute Shortage

#64

Post by gspr »

alt227 wrote: post #33
gspr wrote: post #17

Earlier quoted context omitted.

> GPUs aren’t scarce. There is just a lot of demand, so prices have gone up. What does "scarce" mean to you? I'd wager that to most reasonable people it means "available in a supply that, measured against demand, is low". Market forces typically react to such a state by pushing prices up. Saying "they're not scarce, they're just expensive" is just silly. The exception to this would be during e.g. supply chain hiccups…

The definition of scarce is rare, or insufficient to meet demand. Not just that supply is low in relation to demand, it means that you physically have trouble getting something.

Fine, replace "low" with "too low".

Re: The Growing Compute Shortage

#65

Post by zehaeva »

wkjagt wrote: post #34

When gas prices went up in the 70s because of fuel shortage, smaller (more fuel efficient) cars became more popular to use less fuel to do the same thing. I wonder if the same will happen with compute, by making software more efficient, and do the same thing with less compute.

The only thing I really hope for if this continues for an extended period of time is everyone optimizing their programs to use a minimal amount of ram.

Re: The Growing Compute Shortage

#66

Post by drooby »

This article is about silicon. But the other shortage that matters is meat-compute.

AI is trained on human intelligence. The hyper-scalers are squeezing every last drop of automatically verified reward, and that may get us very far. A compiler passes or fails in milliseconds for free, forever.

But.. "good design taste" has no compiler.

Of course, taste isn't unverifiable. But it's is expensively verifiable. Noisy, slow, and orders of magnitude lower throughput. People with deep domain knowledge often can't articulate well _why_ one design works and the other doesn't. So, judgment arrives as a verdict, and not a crisp rationale. I guess we'll see if sample efficiency outpaces the cost of human judgement.

In the mean time, leverage will sit with whoever holds this tacit knowledge (incumbents). I.e., hospital systems, law firms, chip designers, studios, SaaS that are dominating their niche.. and not with the labs training on it. To me, this is why valuations of companies like Palantir could potentially make sense.

Re: The Growing Compute Shortage

#67

Post by user43928 »

reticulates wrote: post #45
OGWhales wrote: post #38

Earlier quoted context omitted.

> even if we assume their margins are 90% (they are not) How do you know they are not? It will be curious to see the cost of inference for these newly released open weight models and will help give an idea of the actual cost of inference. But for now, I think saying the $200 plans allows for "tens of thousands of dollars worth of inference" provides very little insight when you are measuring the inference cost in API…

We don’t “know” because they haven’t released any numbers but the most optimistic estimates (which many people believe are very very optimistic) put it at 60%: https://newsletter.semianalysis.com/p/anthropic-growth-and-b... The simple question to ask is, if it is so profitable, where is all the money going? If Anthropic have 90% margins on API usage and API usage is $50bn+ in revenue per year, where is the $45bn goin…

I am not knowledgeable about their finances.

But I do wonder how a 60% margin would be realistic when Sonnet costs 3-6x more than GLM 5.2 hosted by third party providers.

Re: The Growing Compute Shortage

#68

Post by cmiles8 »

senordevnyc wrote: post #61
cmiles8 wrote: post #48

Four things are currently correct: 1. There is a huge demand for compute, specifically GPU compute 2. Infrastructure providers are building like crazy, including taking on massive debt to fund this because their own cash flow can’t cover the bills 3. The demand for that compute is broadly being paid for with investor dollars pumping up the valuation of AI companies, not cash flow from said companies. If those subsidi…

Eh, I don’t think there’s any reason to think #3 is true, and the whole thing being a house of cards is predicated on that one. If the massive demand is still present for compute at market rates (which I believe it is), then your second point is just investors spending cap ex to build out valuable and profitable assets, no problem there. Time will tell.

What evidence says otherwise? OpenAI is projected to have massive losses for years to come and all indications are that’s driven by the cost of compute being far higher than the revenue generated by said compute.

Re: The Growing Compute Shortage

#69

Post by dash2 »

reticulates wrote: post #21
loolhahalmao wrote: post #16

Earlier quoted context omitted.

"subsidizing" aka making 30% gross margin instead of 90%.

no, even if we assume their margins are 90% (they are not) they are still losing money because the $200 plans allow for tens of thousands of dollars worth of inference and a huge number of users are milking every cent across multiple accounts. Every “reset” OpenAI and Anthropic do is setting money on fire. If it were true that they’re making money hand over fist they wouldn’t need to raise tens of billions of dollars…

In Amodei’s Dwarkesh podcast he says that they are constantly estimating the increase in demand for the next leg up, then going out to raise money to build for it. So it’s not necessarily that they are raising money because they are unprofitable.

Re: The Growing Compute Shortage

#70

Post by dash2 »

reticulates wrote: post #26
rayiner wrote: post #15

Earlier quoted context omitted.

The compute demand is not fake. Non-coding industries have barely begun to deploy this technology. In the legal sector, I’ve been a tech pessimist my entire career, because it was uniformly quite bad. I’ve spent the last few months demoing legal tools backed by frontier models, and we’re definitely going to buy one of them. They’re real and they work and they address a bunch of needs.

You’re talking across the issue. The demand is real because it is cheap. The demand is being generated by OpenAI and Anthropic selling inference below cost on fixed price plans. If everyone was paying the actual costs then demand would fall through the floor. The legal tools you’re looking at use barely any compute. They’re not driving the compute demand. You can validate this by asking how much they are spending on…

Compute would have to be very expensive not to be cheaper than a junior lawyer.
Post reply on HN