Live data from Hacker News

Something is changing in the unit economics of software

nicolo.xyz

21–30 of 62 posts

Re: Something is changing in the unit economics of software

#21
post #10

Earlier quoted context omitted.

> and so does everything in the software world. Well, no. Copying is free, or so near free it makes zero sense to charge. LLMs are just papering over the damage caused by profit.

Yeah, my point is that LLMs can reach that point too. Especially if you do it clientside. Copying was also way more expensive back in the 60s (accounting just for machine time and electricity, not storage cost, about 100 million times more expensive than today). Everything has a cost.

[deleted]

Re: Something is changing in the unit economics of software

#22
post #10

Earlier quoted context omitted.

> and so does everything in the software world. Well, no. Copying is free, or so near free it makes zero sense to charge. LLMs are just papering over the damage caused by profit.

Yeah, my point is that LLMs can reach that point too. Especially if you do it clientside. Copying was also way more expensive back in the 60s (accounting just for machine time and electricity, not storage cost, about 100 million times more expensive than today). Everything has a cost.

LLMs will never be as cheap or reliable as copying.

Re: Something is changing in the unit economics of software

#23
post #10

Earlier quoted context omitted.

Yeah, my point is that LLMs can reach that point too. Especially if you do it clientside. Copying was also way more expensive back in the 60s (accounting just for machine time and electricity, not storage cost, about 100 million times more expensive than today). Everything has a cost.

LLMs will never be as cheap or reliable as copying.

Definitely, of course. But I think the logic of the author’s article is based on there being a huge difference between the two, or rather a high cost of inference in absolute terms. And that can change and we have seen it change. Which breaks the entire premise of the article going forward, no?

Inference will be always more expensive than db operations or copying, sure. But how much more expensive is the question.

Re: Something is changing in the unit economics of software

#24
If I'm any kind of indicator of where Youtube users are headed, their AI chatbots in the video pages are going to kill their business model.

There are so many videos with hooks/teasers/'you won't believe what we discovered!!1', and now I just pause the video in the first second, ask "what's the tldr" and get the value from the video without a single ad impression (and likely racking up far more opex for Youtube than if I just streamed the video).

Re: Something is changing in the unit economics of software

#25
post #20

>Users began expecting something fundamentally different from software: not just tools that store and retrieve, but products that reason, generate, and respond. Not really. >Every inference call costs money. Not really, either. If you buy your own GPU, rack it, and run an open model, there is no unit cost. This is just expensive hosting infra. You also pay unit costs for SaaS that your software uses (things like SMS…

> If you buy your own GPU, rack it, and run an open model, there is no unit cost. No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.

These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost.

Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.

Re: Something is changing in the unit economics of software

#26
post #7

I think part of the new equation may also become; "Why even pay for the SaaS in the first place if you can just forge the service exactly how you want it?" The benefits of unlimited access to the tool you forge are still there, its just a lot easier to make whatever tool you want. Are there any examples of products containing ai inference that are successful? Products that are beyond just direct access to frontier LL…

As if making all the decisions that go into that product, hosting it, and maintaining it are free! Now do that for all the SaaS you subscribe to. Would you even get any real work done?

There wouldn't be a product. Consumers would just use the underlying foundational models directly to solve their problem. ChatGPT and Claude and Gemini are the products, not your business.

Products are only viable when you do a lot of work to solve a problem, which is then shared by many potential customers. It makes sense to amortize the high costs of solving the problem across all the different customers to reap economies of scale. Businesses then pay for product design, hosting, and maintenance because those costs can also be amortized, and they are cheaper than a bespoke solution for each customer. But if the bespoke solution becomes cheaper than that, because it's generated by an LLM that doesn't need to be paid a living wage, there's no reason to have the product in the first place. Just solve your damn problem and let other people solve theirs.

This is an underrated factor in the market structure of the AI bubble going on now. Anecdotally, we're not seeing new AI-based products other than the foundational models and coding assistants gain traction. Why? Because AI makes it so easy to customize the solution to your particular needs that everybody is just solving their particular needs directly. It's the opposite of the Internet boom, where the network greatly expanded the potential market, reduced the cost of reaching them, and made it economical to spend large amounts of money building a software product that had a TAM of billions. The AI boom instead enables extremely cheap customization, which shrinks the market to a single customer who uses AI to directly solve their problem rather than building a product that's generally applicable.

Re: Something is changing in the unit economics of software

#27
post #9

This is one reason I've been trying to figure out tasks (and products based on those tasks) that can be pushed to the edge, either via small specialized models or small general purpose open models. I suspect the same desire to keep unit costs low is part of why Google is falling behind on the "frontier", but seemingly at the lead, or near it, on models that run on-device. I think they're just focused on making models…

> And, even with specialization and fine-tuning, often a high-end general purpose model is going to do a better job and people don't need a bunch of local tools installed to do their various tasks. I think this is the critical point that would be interesting to see if it holds. Technology seemingly tends towards increased specialization.

> Technology seemingly tends towards increased specialization.

The bitter lesson says the exact opposite.

Re: Something is changing in the unit economics of software

#28
post #11

"Inference" is just software running. It has always cost money to run software, it's just that it is generally too cheap to matter. If a client makes a regular API call to your server, you pay for that compute, probably in the form of a flat hosting fee. If too many calls come in and workload goes up, you pay for a more expensive hosting tier to handle it (or do dynamic scaling which is per-unit of compute). Right no…

> There is a lot of inefficiency right now keeping prices elevated. That will change very fast and soon paying for inference will likely resemble paying for hosting your app.

I don't know about "very fast" or "soon" unless you're speaking in geological terms.

SOTA models like Kimi 3 require thousands of GB of RAM/VRAM to run at speeds that are real-time useful. Manufacturing the memory necessary for that quantity to be available at app-hosting prices will take decades. Software efficiency solutions might drop needed memory by an order of magnitude in that time...but a tenth of an enormous amount is still pretty darn big so won't get us there "soon".

Re: Something is changing in the unit economics of software

#29
post #20

Earlier quoted context omitted.

> If you buy your own GPU, rack it, and run an open model, there is no unit cost. No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.

These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost. Open question wheth…

Good points. Speaking from experience, it’s really hard to make it more economical than using a cloud AI service - even if you utilize the GPU to its fullest. There are, of course, other benefits, such as privacy/control/compliance/security, which should be the real reasons to do this, not cost.

Re: Something is changing in the unit economics of software

#30
post #20

Earlier quoted context omitted.

> If you buy your own GPU, rack it, and run an open model, there is no unit cost. No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.

These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost. Open question wheth…

You do realize that a fully utilised GPU uses about 10 to 50 times more power than an idling one?
Post reply on HN