Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI's economics don't make sense

wheresyoured.at

Post reply on HN
Print view View original

71–80 of 201 posts

Re: AI's economics don't make sense

#71

Post by charcircuit »

Glyptodon wrote: post #39

I think there's another route this goes. At $7k a year or more per eng in token use, I think it's very reasonable to buy engineers machines with obscene GPUs and RAM and run models locally. And if it doesn't make sense now, someone will figure it out and save companies $10k+/eng over 3 years.

That could leave idle time where GPUs are sitting unused. It would be better to have a shared cluster that many engineers all share. And to avoid a cluster not being saturated other companies queries could also be batched. And oh wait we are back to doing AI inference in the cloud as it is an efficient way to serve AI.

Re: AI's economics don't make sense

#72

Post by GaggiX »

zozbot234 wrote: post #47
joshjob42 wrote: post #10

There's a few major problems with the article. The most obvious is that frontier labs are not charging remotely close to the cost of tokens; afaik most estimate north of 80% profit margins. As a reference, providers are profitably providing Kimi K2.6 for $4/1Mtok out. Is that as good as Opus? No, but it's probably at least Sonnet level, so that's ~4x cheaper than Sonnet while still being profitable to serve on the ma…

It's not even a fixed cost per token (even though it's billed that way, and that's still miles better than a fixed-price all you can eat). You're incurring a cost that's proportional to generated tokens times the context for each (plus the prefill cost for any uncached input), so the expense grows quadratically with your average generated context. This all becomes extremely visible when trying to do agentic coding wi…

Using larger contexts often costs more in the APIs or consume more of your quota but this is becoming less of a problem with models using more clever attention mechanisms and not just full attention on all layers.

You can look at: https://sebastianraschka.com/llm-architecture-gallery/ and see how much things have changed.

Re: AI's economics don't make sense

#73

Post by xienze »

bwestergard wrote: post #48
loeg wrote: post #41

Earlier quoted context omitted.

> How many tokens can you realistically burn through in one chat session? I've used single digit billions in a couple days, FWIW.

What sort of work were you doing?

Not the parent, but the way developers are basically trying to create entire development "teams" consisting of multiple agents that work around the clock using the latest, most expensive models (naturally) lends itself to burning insane amounts of tokens.

Re: AI's economics don't make sense

#74

Post by CamperBob2 »

pmdr wrote: post #68

I wonder how long until this post is flagged/"shadowbanned". Such was the fate of almost all of Ed's posts on HN, with little surprise as to why.

People who don't adjust their prior outlook in light of newer data may not be the best fit around here. I'm OK with that.

Re: AI's economics don't make sense

#75

Post by OrvalWintermute »

I think the company Taalas alone destroys Ed’s arguments

Because, comparing vs GPUs

~16k–17k tokens/second per user

10x power efficiency

20x cheaper production

Model to Si ~ 60 to 90 days

We have every reason to believe SW_to_Si will facilitate improving economics

Re: AI's economics don't make sense

#76

Post by tuveson »

JohnMakin wrote: post #53

I've sort of lost some respect for ed that I had early on in the hype cycle - he's still right about some things, but I can see him slowly and subtly retreating from his strong position, held even a few months ago, that these things will never ever be useful for anything and it's all a scam because they don't actually do anything at all except burn money. He would say it like 8 times a monologue. I remember one podca…

I’m remember when CrowdStrike caused that huge outage, he basically blamed Windows / Microsoft for it. I kind of stopped taking him seriously after that. I more-or-less agree with his point of view, but he seems more interested in selling outrage rather than journalism.

Re: AI's economics don't make sense

#77

Post by BosunoB »

All subscription models are subsidized by users who don't use much. The fact that somebody on a $20 sub might get $50 in value isn't crazy if there are 3 people who only get $10 in value. This isn't some sign that the model is broken, it's the intended outcome.

Also, I didn't read this whole thing, but I have yet to see Zitron respond to the strongest AI financials claim, which is that the models themselves are profitable on a life-cycle basis, even if the companies are not profitable on an annual basis due to capital expenditure. Dario made this claim exactly, and it more or less blows all of Zitron's financials arguments up.

Re: AI's economics don't make sense

#78

Post by Maxatar »

ToucanLoucan wrote: post #18
joshjob42 wrote: post #10

There's a few major problems with the article. The most obvious is that frontier labs are not charging remotely close to the cost of tokens; afaik most estimate north of 80% profit margins. As a reference, providers are profitably providing Kimi K2.6 for $4/1Mtok out. Is that as good as Opus? No, but it's probably at least Sonnet level, so that's ~4x cheaper than Sonnet while still being profitable to serve on the ma…

> That said, even if a developer is burning $50/hr, many, many employees at large companies cost more than $100k/yr to employ all costs considered, so making them say 20-30% more productive can easily make that worth it for most. If the labs shave their margins ultimately to more like 20-30%, you'd have ~$15/hr in costs to use the services, and nearly every white collar job is way over 30k/yr to employ. If your salar…

You wrote an entire wall of text when you could have just taken 10 seconds to review what you call the "most basic back-of-napkin math" and realized you were off by two and a half orders of magnitude.

Re: AI's economics don't make sense

#79

Post by CodingJeebus »

hparadiz wrote: post #61
JohnMakin wrote: post #53

I've sort of lost some respect for ed that I had early on in the hype cycle - he's still right about some things, but I can see him slowly and subtly retreating from his strong position, held even a few months ago, that these things will never ever be useful for anything and it's all a scam because they don't actually do anything at all except burn money. He would say it like 8 times a monologue. I remember one podca…

The economics is spending a few hundred bucks on software for an IC you're already paying over ten grand a month in order to make them more productive. How are supposedly smart industry experts not seeing this obvious fact? Are these guys actually experts?

It's a few hundred bucks per month for now, but that's not going to last. At some point, the industry is going to pivot towards tracking token-based productivity because it's not going to be cheap forever unless FOSS models catch up.

Re: AI's economics don't make sense

#80

Post by JohnMakin »

tuveson wrote: post #76
JohnMakin wrote: post #53

I've sort of lost some respect for ed that I had early on in the hype cycle - he's still right about some things, but I can see him slowly and subtly retreating from his strong position, held even a few months ago, that these things will never ever be useful for anything and it's all a scam because they don't actually do anything at all except burn money. He would say it like 8 times a monologue. I remember one podca…

I’m remember when CrowdStrike caused that huge outage, he basically blamed Windows / Microsoft for it. I kind of stopped taking him seriously after that. I more-or-less agree with his point of view, but he seems more interested in selling outrage rather than journalism.

I agree. Early on, it felt more like journalism, then I think he blew up and found something that works. If you challenge him on this, he will call you insecure or jealous, which I also find obnoxious[0]. I also find it highly ironic that all the ads on his podcast, at least on apple, are selling AI related products.

[0] - https://www.reddit.com/r/BetterOffline/comments/1p5zv33/why_...

Post reply on HN