Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI's Affordability Crisis

blog.dshr.org

Post reply on HN
Print view View original

281–290 of 436 posts

Re: AI's Affordability Crisis

#281

Post by VorpalWay »

akomtu wrote: post #255
Legend2440 wrote: post #217

Earlier quoted context omitted.

GPUs are not really the ideal architecture for running neural networks; they are heavily bottlenecked by memory bandwidth and struggle to keep all their tensor cores supplied with data. There is significant room to make more specialized neural network accelerators with new compute-in-memory architectures. If the brain can run 86 billion neurons on 30W it must be possible.

Our brains run 86 billion neurons the same way a waterfall runs a fluid simulation with N quadrillion particles.

There are already some companies doing specialised inference hardware, Cerebras Systems for example. Such designs are still early days and I wouldn't be surprised to see more innovation there. Though because custom silicon design takes time I expect a multi-year cycle.

For training, not sure. But even if training runs on GPUs, once you have the model the main cost is inference.

Re: AI's Affordability Crisis

#282

Post by Analemma_ »

andrekandre wrote: post #195
827a wrote: post #71

> Zitron's numbers don't tell us the real cost of generating tokens but, subject to the assumption that the platforms are not subsidizing the token price, that means Anthropic is subsidizing their enterprise customers by up to 40 times, and OpenAI up to 70 times Neither Anthropic nor OpenAI are subsidizing enterprise customers. Neither Anthropic nor OpenAI allow Business nor Enterprise customers access to the high va…

> Neither Anthropic nor OpenAI allow Business nor Enterprise customers access to the high value $200/mo plan. they may not "allow" it, but i've seen first hand enterprises encourage employees to use these accounts personally and get reimbursed later to avoid pay-as-you-go w/limits pricing for users who do tokenmaxing as a cost control measure...

Do these companies know they're letting their source code (and internal documentation, chats, communications, etc.) get used as training data? On the flat-rate consumer subscriptions, everything you send is fair game to train on, which is not true of the enterprise plans.

Shows how much they value their IP, I guess.

Re: AI's Affordability Crisis

#283

Post by whimsicalism »

androiddrew wrote: post #278

The over investment by VC means that yeah, they are offering all of this below market rate. It's like Enron where they have to keep the scheme going, and dumping on retail investors is the only thing they can do now. So we are going to go through a big IPO period. Everything will fall apart because VCs already extracted the growth value, and that will show up after the bag has been passed. Things will implode. What s…

When we say below market rate, what do we mean? The token economics are definitely such that they are charging more than it costs to serve these models with reasonable assumptions on param/activated size.

Re: AI's Affordability Crisis

#284

Post by VorpalWay »

akazantsev wrote: post #164
jcgrillo wrote: post #81

Earlier quoted context omitted.

There's a secret third option: learn. At one point, all of us were "nontechnical", but we learned. The trick is to never stop.

Is it? Learning is one thing. But owning a large codebase, you see for the first time, is a completely different level.

When I started working on a large legacy C++ code base in the early 2010s, I learnt. It took probably close to a year until I was proficient enough that I didn't regularly need to ask where to fins things. This is a skill we used to have (and some of us still have).

Though if your code base is all a vibe coded mess and you don't have a senior human colleague to ask... Good luck?

Re: AI's Affordability Crisis

#285

Post by shaewest »

gizzlon wrote: post #140

> Sales and Marketing: $5.73 billion .. That is, OpenAI spent 44% of their revenue on sales and marketing! Anyone know what they are spending this on? Can't remember seeing one OpenAI ad.. Is it just pr and influencers? Ads in the US?

I've easily gotten (low) hundreds of OpenAI youtube ads. More recently they've been pushing 'Free OpenAI Image generation' to me, in the past they pushed Codex more, but I have a sub for that now, so I guess it works.

Re: AI's Affordability Crisis

#286

Post by HDThoreaun »

surgical_fire wrote: post #274
HDThoreaun wrote: post #246

Earlier quoted context omitted.

Well the third party operators on open router are assuredly operating at a profit, including depreciation. The only reason they’d be profitable at 1/10 the price of the labs while the labs aren’t profitable is if inference costs the labs 10x as much per token

> Well the third party operators on open router are assuredly operating at a profit, including depreciation They don't train new models. They have to depreciate their GPUs, which I hope they do.

I agree inference probably isn’t profitable yet if you include training costs. My claim is that marginal revenue from inference is higher than marginal cost. If that’s the case then if they scale enough the training costs will be amortized.

I realize I said assuredly when I meant assumedly. My mistake. I agree it’s possible that the third party open model hosters aren’t actually profitable, my claim does rest on the opposite.

Re: AI's Affordability Crisis

#287

Post by whimsicalism »

woeirua wrote: post #218

It's not an affordability crisis, it's a financial crisis. The models get cheaper super fast. By this time next year Fable 5 will cost less than Sonnet does today. That's not the problem. The problem is that many companies are going to realize that they don't get any ROI from AI. Generating code faster != more profit. Most of the Fortune 500 will likely realize this and then the token budgets will come crashing down.…

Do they get ROI from additional workers? Honestly it is difficult for me to imagine white collar companies that have positive marginal profit of labor but not positive marginal profit of tokens, especially in the future.

Re: AI's Affordability Crisis

#288

Post by whimsicalism »

simianwords wrote: post #247
woeirua wrote: post #218

It's not an affordability crisis, it's a financial crisis. The models get cheaper super fast. By this time next year Fable 5 will cost less than Sonnet does today. That's not the problem. The problem is that many companies are going to realize that they don't get any ROI from AI. Generating code faster != more profit. Most of the Fortune 500 will likely realize this and then the token budgets will come crashing down.…

Its very interesting how you are contradicting the whole article's axioms and then arriving at the same conclusion that we are in for a crash! Rational takeaway is to step back and analyse what's really happening here. - Are we really in for a crash? - What does it say about the culture and people's mental models that we have two radically opposing viewpoints on AI costs and people still arrive at same conclusion?

It's just schadenfreude/ressentiment. Our societal values are very christian even as we secularize and you can see this reflected everywhere you turn. Same thing with people feeling there ought to be a 'catch' to GLP1s, etc. etc.

Re: AI's Affordability Crisis

#289

Post by qurren »

steveBK123 wrote: post #44

I think the biggest problem is not necessarily the cost to develop & serve the models, but how quickly user behavior changed with token based pricing. I know a lot of people at companies where the marching orders changed on a dime end of Q1/start of Q2. These are shops that were fully on the "use AI or die (because we will fire you)" train. Now there's monitoring, reporting, alerting not just on overall cost but on "…

What I don't understand is why some companies are so stingy on AI token usage.

If you're paying an engineer $X and they're getting 3x the amount of work done you should be happy paying up to $2X in AI tool usage.

In reality many companies start complaining at their employees when they hit $0.1*X or less.

Re: AI's Affordability Crisis

#290

Post by luke5441 »

burningChrome wrote: post #176
steveBK123 wrote: post #44

I think the biggest problem is not necessarily the cost to develop & serve the models, but how quickly user behavior changed with token based pricing. I know a lot of people at companies where the marching orders changed on a dime end of Q1/start of Q2. These are shops that were fully on the "use AI or die (because we will fire you)" train. Now there's monitoring, reporting, alerting not just on overall cost but on "…

I can give you some additional anecdotal evidence to support your comment. I work at a Fortune 200 company. At first, it was the Wild West. Need an LLM? You got it. Need to or want to build an army of agents? Done and done. We literally had everything at the tips of fingers for about 3 months. Teams were building their own internal tools, the team I work on canceled contracts with several software vendors because tea…

Maybe it was your company with the 500 million Claude bill.
Post reply on HN