Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI's Affordability Crisis

blog.dshr.org

Post reply on HN
Print view View original

331–340 of 436 posts

Re: AI's Affordability Crisis

#331

Post by bluGill »

whimsicalism wrote: post #291
steveBK123 wrote: post #44

I think the biggest problem is not necessarily the cost to develop & serve the models, but how quickly user behavior changed with token based pricing. I know a lot of people at companies where the marching orders changed on a dime end of Q1/start of Q2. These are shops that were fully on the "use AI or die (because we will fire you)" train. Now there's monitoring, reporting, alerting not just on overall cost but on "…

I think this panic around overuse is largely just managers talking amongst themselves and everyone trying to see what everyone else is doing. It is frankly hard to interpret in any other way

Also talking to accountants who will sound alerts if the bottom line isn't looking good enough.

Re: AI's Affordability Crisis

#332

Post by steveBK123 »

brandensilva wrote: post #307
steveBK123 wrote: post #293

Earlier quoted context omitted.

Most companies are not seeing 2x or 3x of value produced from developers with AI is the reason

Yeah the cost doesn't justify the value. Spending x3 on tokens is meaningless if they aren't seeing the profit side scale with it. Going faster on product features doesn't mean it translates to more money. In fact I'm not even sure some customers can handle the speed at which a team could move with feature delivery. So scaling horizontally in different markets seems like an advantage if a product is already mature. W…

> Going faster on product features doesn't mean it translates to more money.

Exactly this. Unless the new features directly drive new users or new revenue from existing users, for many products iterating 2x faster does not mean 2x ROI.

Additionally, from anecdotes here, at work, and in my network.. a lot of the unlocked developer velocity is going to fun/frivolous/extra things. I think part of it is developers have their own features they want for themselves that are the easiest and most direct thing to deploy LLMs against, in the absence of good direction.

Yes it's cool you finally achieved 100% test coverage, or you wrote a new utility that makes your job easier, or cleared the 2 year old ticket that was 100 deep in your backlog, etc. But there were ROI reasons these things were not done previously.

That's already putting aside the fact that development going 3x faster doesn't increase end to end output by 3x because <100% of a SWEs job is development.

Re: AI's Affordability Crisis

#333

Post by bluGill »

827a wrote: post #279
andrekandre wrote: post #195

Earlier quoted context omitted.

> Neither Anthropic nor OpenAI allow Business nor Enterprise customers access to the high value $200/mo plan. they may not "allow" it, but i've seen first hand enterprises encourage employees to use these accounts personally and get reimbursed later to avoid pay-as-you-go w/limits pricing for users who do tokenmaxing as a cost control measure...

Yeah I just mean that if a business came to them and asked for fifty licenses to the $200/mo plan, OpenAI would tell them to kick dirt and basically pay API pricing. Startups should 100% just be telling their employees they can expense up-to $whatever/mo in AI-related expenses, and let software engineers go buy personal Codex/Claude subscriptions.

Many large companies won't allow employees to expense such things personally. It is too easy to allow other fraud if that is allowed.

Though large companies will demand limits if API or whatever they can get is too expensive.

Re: AI's Affordability Crisis

#334

Post by steveBK123 »

stingraycharles wrote: post #266
burningChrome wrote: post #200

Earlier quoted context omitted.

AFAIK nobody was collecting analytics. The one team I was working on had put out a goal of "30% more efficient" using AI tools. Its about as subject as you can get. We never got around to what exactly that meant before everything got shut down. Myself and several other devs were laughing about the whole thing. The company was so amped about what AI could do they never even bothered collecting any analytics that would…

> Even some of my team members were talking about the placebo effect AI has had on a lot of C-Suite folks. What do you mean placebo effect? They thought things were created with AI while they actually weren’t?

For a lot of C-suite there has been more of a fast-follow approach to AI. CEO hears 3rd hand what competitors are doing, tells his CTO to "do AI" and "hire a head of AI".

So a lot of motion (do AI) was created without a destination (product outcome). The motion was what was being measured (we are doing AI).

Re: AI's Affordability Crisis

#335

Post by bluGill »

gizmo686 wrote: post #105
tacone wrote: post #27

My take is that Anthropic and OpenAI simply are NOT competing on price. 2 big players are often not enough to create tension on price. Chinese models and open model providers are, indeed, competing on price, and the difference shows.

1 player is enough to create tension on price when "don't buy it at all" is a comptetative option. By most accounts, Anthropic and OpenAI both lose to "just don't buy" when they try charging at cost.

Nobody has seen the financials who is talking. We have various rumors of costs but no real reason to believe any of them.

Re: AI's Affordability Crisis

#336

Post by Gigachad »

zhivota wrote: post #330
Gigachad wrote: post #265

Earlier quoted context omitted.

The accounting is all out of whack with these companies. If a GPU had a useful lifespan of around 3 years than it’s more a constant expense than R&D. They want you to believe they can cut the R&D at any time and be profitable but it’s likely they would completely collapse without that spend.

I don't see why 3 years is the right number there. I'm using a 10 year old model GPU to generate tokens locally, and given the bottlenecks, a commercial model focused on RAM and transfer speeds should have a longer depreciation curve than 3 years.

Because these GPUs in data centers chew power and take up space. If in 3 years there is a new model that processes far more tokens with the same power and time the economics quickly say the hardware is cheaper to replace than to continue running.

As a hobbiest at home the numbers are different and you can afford to do something inefficient.

Re: AI's Affordability Crisis

#337

Post by jwrallie »

Not a heavy user but I got a feeling for this early since I have a Copilot subscription I get for free as an educator.

I used in a day or two the limit that would last me a month. Downgrading from Sonnet to Gemini Flash was the only way to keep the limit longer, and who knows when cheaper models will be discontinued for something more expensive.

I don’t know if the prices will remain low, but at least Chinese models being open make them have no control over when it is discontinued, I think learning to work with open models is a good direction, even if not running it on your own hardware.

Re: AI's Affordability Crisis

#338

Post by Gigachad »

mkozlows wrote: post #300
Gigachad wrote: post #258

Earlier quoted context omitted.

Our company went from “AI AI AI” to “GitHub Copilot has been suspended due to exceeding the budget” with this month’s price increase.

If your company was giving you Copilot, they were never that far on board the AI train anyway.

Github Copilot is a different product than the Windows / Office one. Horrible naming I know but it's basically just a UI and router to pick whatever model you want.

Re: AI's Affordability Crisis

#339

Post by pchristensen »

bluefirebrand wrote: post #318
mkozlows wrote: post #302

Earlier quoted context omitted.

10% of a developer's cost is something like $4000/month. Many companies are complaining at a point that's well, well below that. (I think they are being irrational, and that the mental model they have of AI costs -- "how much are we spending on tooling for this developer?" -- is going to shift over time to something more sensible, but those kinds of short-sighted companies are the ones that are having cost panics.)

> 10% of a developer's cost is something like $4000/month What company pays developers $40000/month and are they hiring?

A company's fully loaded cost (payroll taxes, insurance, fixed company costs, infra, real estate, etc) is usually about double the employee's salary. Half of $40k is 20k/mo, which is ~$250k/hr. That's a lot but very in-bounds for software engineering salaries.

Re: AI's Affordability Crisis

#340

Post by bluGill »

lbreakjai wrote: post #321
travisb wrote: post #158

I think a lot of the cost comparisons to employees are off by a factor of 2 or more. AI is the ultimate contractor. Available instantly. Doesn't charge during idle periods. Pre-vetted and pre-trained. No contract negotiations or complex accounting. That is worth a small multiple of the fully-loaded employee cost. So AI might be easily worth more than $200 per human-equivalent hour. With high utilization, that might b…

AI is more like a union that controls the entire labour pool. I'm one out of a few million developers. I've got very limited bargaining power. I can't withdraw my labour because I need it to make rent and buy food. My cost has a very predictable ceiling. On the other hand, there's two AI labs, that could afford to eat your profit, because what are you gonna do? They're your entire labour force.

There are more than two options. The others are not as good, but depending on who you believe they might be good enough.
Post reply on HN