Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

91–100 of 319 posts

Re: LLMs are cheap

#91

Post by hajile »

deadbabe wrote: post #19

Given how addicted people are to using LLMs steep price hikes are almost certainly guaranteed at some point. When this happens, what we will see is once again the rich and privileged will benefit from the use of LLMs while the poor have to just rely on their own brains. Consider how some students will have to grow up struggling through school without any LLMs while rich kids breeze their way through everything with t…

If they are dividing a few billion dollars in model training between a small number of rich people, it quickly becomes too expensive even for them.

Meanwhile, a free model running locally is good enough for most people. This causes pricing pressure (and I think is probably going to bankrupt most of the AI companies).

More likely IMO is that AI becomes a loss-leader. It'll all be stuff like Grok or DeepSeek where the real profit is in censorship and propaganda.

Re: LLMs are cheap

#92

Post by jsnell »

physicsguy wrote: post #69

LLMs are heavily subsidised. If you self-host them and run them at cost, then you find that the GPU costs are high, and that's largely without the additional tools that OpenAI and Anthropic provide and which also must cost a lot to operate.

If you self-host, you likely won't have anywhere near enough volume to do efficient batching, and end up bottlenecked on memory rather than compute.

E.g. based on the calculations in https://www.tensoreconomics.com/p/llm-inference-economics-fr..., increasing batch size from 1 to 64 cuts the cost per token to 1/16th.

Re: LLMs are cheap

#95

Post by jarym »

iamnotagenius wrote: post #32
pama wrote: post #24

Earlier quoted context omitted.

Please read the DeepSeek analysis of their API service (linked in this article): they have 500% profit margin and they are cheaper than any of the US companies serving the same model. It is conceivable that the API service of OpenAI or Anthropic have much higher profit margins yet. (GPUs are generally much more cost effective and energy efficient than CPU if the solution maps to both architectures. Anthropic certainl…

With all due respect to Deepseek, I would take their numbers with grain of salt, as they might as well be politically motivated.

Any more politically motivated than a model from anywhere else?

Re: LLMs are cheap

#96

Post by adrian_b »

brookst wrote: post #74
WhyIsItAlwaysHN wrote: post #65

There's something I don't get in this analysis. The queries for the LLM which were used to estimate costs don't make a lot of sense for LLMs. You would not ask an LLM to tell you the baggage size for a flight because there might be a rule added a week ago that changes this or the LLM might hallucinate the numbers. You would ask an LLM with web search included so it can find sources and ground the answer. This applies…

Just tried asking “what is the maximum carryon size for an American Airlines flight DFW-CDG” and it used a webs search, provided the correct answer, and provided links to both the airline and FAA sites. Why wouldn’t I use it like this?

I do not see which is the added benefit provided by the LLM in such cases, instead of doing yourself that web search, and for free.

Re: LLMs are cheap

#97

Post by graemep »

fedeb95 wrote: post #46

I don't think LLMs are inherently "costly" or "cheap". This doesn't really matter. Gold is pricey, but its usages justify the cost. Will LLMs, as they are used and evangelized now, have a true positive return for those using it? In some domains it will, most probably not everywhere and not for everyone.

> Gold is pricey, but its usages justify the cost

I understand the point, but gold is expensive because it is a traditionally agreed store of value, rather than because of its usage. Rhodium would be a better example.

Re: LLMs are cheap

#98

Post by mmcnl »

I think this is a good analysis but falls a little short. Sure, the price is not high for inference, but what about the cost? To be fair, the author already tries to answer this claim, but you could look more critically at this question. Something like: taking into account the insane amount of capital that is being spent and injected into AI companies, what is the strategy to break-even in a reasonable amount of time? What would be the implications for the price over time from now on? That's an interesting thought experiment that, at least in my head, raises the question if the price we're paying for inference today is actually fair.

Re: LLMs are cheap

#99

Post by WithinReason »

iamnotagenius wrote: post #32
pama wrote: post #24

Earlier quoted context omitted.

Please read the DeepSeek analysis of their API service (linked in this article): they have 500% profit margin and they are cheaper than any of the US companies serving the same model. It is conceivable that the API service of OpenAI or Anthropic have much higher profit margins yet. (GPUs are generally much more cost effective and energy efficient than CPU if the solution maps to both architectures. Anthropic certainl…

With all due respect to Deepseek, I would take their numbers with grain of salt, as they might as well be politically motivated.

is that better or worse than commercially motivated?

Re: LLMs are cheap

#100

Post by jbd0 »

physicsguy wrote: post #69

LLMs are heavily subsidised. If you self-host them and run them at cost, then you find that the GPU costs are high, and that's largely without the additional tools that OpenAI and Anthropic provide and which also must cost a lot to operate.

Before I started self-hosting my LLMs with Ollama, I imagined that they required a ton of energy to operate. I was amazed at how quickly my local LLM operates with a relatively inexpensive GeForce RTX 4060 with 8GB VRAM and an 8b model. The 8b model isn't as smart as the hosted 70b models I've used, but it's still surprisingly useful.
Post reply on HN