Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs are cheap

snellman.net

Post reply on HN
Print view View original

111–120 of 319 posts

Re: LLMs are cheap

#111

Post by ceejayoz »

brookst wrote: post #74
WhyIsItAlwaysHN wrote: post #65

There's something I don't get in this analysis. The queries for the LLM which were used to estimate costs don't make a lot of sense for LLMs. You would not ask an LLM to tell you the baggage size for a flight because there might be a rule added a week ago that changes this or the LLM might hallucinate the numbers. You would ask an LLM with web search included so it can find sources and ground the answer. This applies…

Just tried asking “what is the maximum carryon size for an American Airlines flight DFW-CDG” and it used a webs search, provided the correct answer, and provided links to both the airline and FAA sites. Why wouldn’t I use it like this?

That search query brings up https://www.aa.com/i18n/travel-info/baggage/carry-on-baggage... for the first result, which says "The total size of your carry-on, including the handles and wheels, cannot exceed 22 x 14 x 9 inches (56 x 36 x 23 cm) and must fit in the sizer at the airport."

What benefit did the LLM add here, if you still had to vet the sources?

Re: LLMs are cheap

#112

Post by leeoniya »

WithinReason wrote: post #99
iamnotagenius wrote: post #32

Earlier quoted context omitted.

With all due respect to Deepseek, I would take their numbers with grain of salt, as they might as well be politically motivated.

is that better or worse than commercially motivated?

commercial motivatation needs to show eventual profit to be sustainable, while political does not.

though at the outset (pre-profit / private) it's hard to say there's much difference.

Re: LLMs are cheap

#113

Post by lumost »

xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

We don’t know what the marginal cost of inference is yet however. So far, users are demonstrating that they are willing to pay more for LLMs than traditional web experiences.

At the same time, cards have gotten >8x more efficient over the last 3 years, inference engines >10x more efficient and the raw models are at least treading water if not becoming more efficient. It’s likely that we’ll lose another 10-100x off the cost of inference in the next 2 years.

Re: LLMs are cheap

#114

Post by Havoc »

Guess it comes down to how heavy the query is in context size. If you’re not doing RAG and instead just inlining large amounts then it won’t stay cheap.

But yeah 0.20 per million is nothing for light use.

Re: LLMs are cheap

#115

Post by JimDabell »

adrian_b wrote: post #96
brookst wrote: post #74

Earlier quoted context omitted.

Just tried asking “what is the maximum carryon size for an American Airlines flight DFW-CDG” and it used a webs search, provided the correct answer, and provided links to both the airline and FAA sites. Why wouldn’t I use it like this?

I do not see which is the added benefit provided by the LLM in such cases, instead of doing yourself that web search, and for free.

I just tried that search on Google.

The first thing I saw was the AI summary. Underneath that was a third-party site. Underneath that was “People also ask” with five different questions. And then underneath that was the link to the American Airlines site.

I followed the line to the official site. I was presented with a “We care about your privacy” consent screen, with four categories.

The first category, “Strictly necessary”, told me it was necessary for them to share info with eleven entities, such as Vimeo and LinkedIn, because it was “essential to our site operation”.

The remaining categories added up to 59 different entities that American Airlines would like to share my browsing data with while respecting my privacy.

Once I dismissed the consent screen, I was then able to get the information.

Then I tried the question on ChatGPT. It said “Searching the web”, paused for a second, and then it told me.

Then I tried it on Claude. It paused for a second, said “Searching the web”, and then it told me.

Then I tried it on Qwen. It paused for a second, then told me.

Then I tried it on DeepSeek. It paused for a second, said “Searching the web”, and then it told me.

All of the LLMs gave me the information more quickly, got the answer right, and linked to the official source.

Yes, Google’s AI answer did too… but that’s just Google’s LLM.

Websites have been choosing shitty UX for decades at this point. The web is so polluted with crap and obstacles it’s ridiculous. Nobody seems to care any more. Now LLMs have come along that will just give you the info straight away without any fuss, so of course people are going to prefer them.

Re: LLMs are cheap

#116

Post by xpe »

TZubiri wrote: post #12

"Data from paid API queries will also typically not be used for training or tuning the models, so getting access to more data wouldn't explain it." Source? Is this in the API ToS?

> Data from paid API queries will also typically not be used for training or tuning the models...

Extremely unlikely in my opinion. I would expect some forms of customer data are used for some kind of value or competitive advantage. If not used outright, this might still include transformed, summarized, aggregated, or anonymized data. In my view, various mappings from legal terms of service to the myriad ways data can be massaged leads to massive gray areas. Expecting this to tilt in favor of customer privacy does not match historical practice nor incentives.

Re: LLMs are cheap

#117

Post by Sevii »

Workaccount2 wrote: post #106
xxbondsxx wrote: post #5

You can't compare an API that is profitable (search) to an API that is likely a loss-leader to grab market share (hosted LLM cloud models). Sure there might not be any analysis that proves that they subsidized, but you also don't have any evidence that they are profitable. All the data points we have today show that companies are spending an insane amount of capex on gaining AI dominance without the revenue to achiev…

Just wait till there are ads for free users, which is going to happen. Depending on how insidious these ads are, they could be extremely profitable too, like recommending products and services directly in context.

They could dynamically update the system prompt with ad content on a per request basis. Lots of options.

Re: LLMs are cheap

#118

Post by [deleted] »

dale_glass wrote: post #58
paxys wrote: post #20

The entire comparison hinges on people only making simple factual searches ("what is the capital of USA") on both search engines and LLMs. I'm going to say that's far enough from the standard use case for both these sets of APIs to be entirely meaningless. - If I'm using a search engine, I want to search the web. Yes these engines are increasingly providing answers rather than just search results, but that's a UI/pro…

I think the issue is that the classical search engine model has increasingly become less useful. There's less experts using search engines. Normal people treat search engines less like an index search and more like a person. Asking an old school search engine "What is the capital of USA" is actually not quite right, because the "what is" is probably quite superfluous, and you're counting on finding some sort of educa…

[deleted]

Re: LLMs are cheap

#119

Post by eschatology »

I would add a small asterisk that a given sentence may result in different number of tokens depending on the model and the tokenization method they use, so it’s unfortunately not as straightforward to get the precise dollar value for a given input.

Re: LLMs are cheap

#120

Post by jsnell »

TZubiri wrote: post #12

"Data from paid API queries will also typically not be used for training or tuning the models, so getting access to more data wouldn't explain it." Source? Is this in the API ToS?

OpenAI: https://platform.openai.com/docs/guides/your-data

> As of March 1, 2023, data sent to the OpenAI API is not used to train or improve OpenAI models (unless you explicitly opt in to share data with us).

Anthropic: https://privacy.anthropic.com/en/articles/7996868-is-my-data...

> By default, we will not use your inputs or outputs from our commercial products to train our models.

> If you explicitly report feedback or bugs to us (for example via our feedback mechanisms as noted below), or otherwise explicitly opt in to our model training, then we may use the materials provided to train our models.

Google: https://ai.google.dev/gemini-api/terms#paid-services

> When you use Paid Services, including, for example, the paid quota of the Gemini API, Google doesn't use your prompts (including associated system instructions, cached content, and files such as images, videos, or documents) or responses to improve our products

Post reply on HN