Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs put the free web at risk

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 22 posts

LLMs put the free web at risk

#1

Post by anfernan »

My cofounder and I think the rise of AI has put the free web at risk.

Despite their claims, ChatGPT and Perplexity don’t provide live or complete information from the internet. Instead, they index only the parts of webpages their algorithms deem important. These limitations have given rise to a new breed of AI browser agents powered by the same foundational LLM tech.

These LLMs + agents will likely account for over 90% of web traffic in the near future. Bots scrape content without compensating websites for it. As humans access the web through LLMs and agents more, they directly view websites less. The loss of ad revenue from human visitors threatens many websites’ ability to provide their content publicly for free.

We’ve been considering solutions to these problems, and want to know what you think. 1) Are they problems? 2) How big and serious are they? 3) What are possible solutions?

We have some ideas…

Re: LLMs put the free web at risk

#3

Post by tpswa »

Depends on the site! If you're trying to sell products, you don't necessarily care so long as your products are discoverable by ChatGPT/Perplexity/an agent -- visibility/discoverability is definitely a priority for some businesses. If you're selling ads it definitely is a risk, though licensing deals are one solution but that seems to be only available currently for bigger companies.

Re: LLMs put the free web at risk

#4

Post by anfernan »

tpswa wrote: post #3

Depends on the site! If you're trying to sell products, you don't necessarily care so long as your products are discoverable by ChatGPT/Perplexity/an agent -- visibility/discoverability is definitely a priority for some businesses. If you're selling ads it definitely is a risk, though licensing deals are one solution but that seems to be only available currently for bigger companies.

Definitely! If your revenue comes from selling something to site visitors, you want to be scraped. If you're selling ads, (as far as we know) your options are to either offer an API or make individual licensing agreements. There's no standardized, go-to place to do this.

Re: LLMs put the free web at risk

#5

Post by paulcole »

> My cofounder and I think the rise of AI has put the free web at risk

To paraphrase Upton Sinclair, “When your paycheck depends on you thinking something, it will be very hard to get you to not think it.”

Re: LLMs put the free web at risk

#6

Post by mattbrewsbytes »

I think human written, compelling, deep content will survive. Fluff content sites will likely be squashed by search engines favoring brands and authority over linkage stats because there is so much AI generated content that's mediocre, the search engines have to weed thru that to have relevant results.

Often now though before you see results, you see AI summaries. This is perfectly fine if someone is casually searching something like ratio of oil and vinegar for salad dressing. That person likely is making a salad right now and needs to know and does not care about reading through 5 screens of scrolling content, ads, popups, etc. to get a recipe.

Now if someone is researching the history of salad dressings and wants to go deep into types of oils, vinegars, mixture ratios, pairing with different lettuce types, fruits, nuts, etc. having content that provides that level of information is helpful, but to a smaller audience. That audience is more likely to click links to products or ads than someone just looking for a ratio right now.

The human impression counts for content sites is likely to go down, a lot for fluff content, but those sites are likely losing impressions from people that weren't clicking the ads or buying products anyhow.

I'm not in advertising/marketing so I don't know what impacts that has ... maybe for some time period, ad revenues will go down because sites lose the fluff visitors and number of visitors carries weight in advertising processes, right? Then maybe the processes catch up and realize that the good content sites have higher "stickyness" factor so conversion rates and length of users visit are much more important than page impression counts?

Re: LLMs put the free web at risk

#7

Post by itsankur »

mattbrewsbytes wrote: post #6

I think human written, compelling, deep content will survive. Fluff content sites will likely be squashed by search engines favoring brands and authority over linkage stats because there is so much AI generated content that's mediocre, the search engines have to weed thru that to have relevant results. Often now though before you see results, you see AI summaries. This is perfectly fine if someone is casually searchi…

Yeah I mean the salad example is great because in that case it feels ridiculous to have to look at ads.

Maybe we don’t need to protect websites that rely on advertised content? Maybe it makes a better internet?

Based on what you’re saying here it sounds like good content will inevitably go behind paywalls in this case, and the number of paid subs people need to have will only continue to explode. Aka no free internet. Maybe that was bound to happen one way or another.

And all LLM providers will pay their data providers directly by API or something?

original internet (DARPA) was meant for communicating research and science. Maybe we return to that being the primary thing that will continue to be free?

Re: LLMs put the free web at risk

#9

Post by scarface_74 »

Getting rid of ad supported content and SEO optimizations? Nothing of value will be lost. There is nothing “free” about the tracking and evilness of adtech.

I have been paying $20/month for ChatGPT with web search just so I could avoid sites like yours.

The first thing I see when I go to your site is a half page cookie banner that wants me to accept tracking.

Re: LLMs put the free web at risk

#10

Post by kjok »

scarface_74 wrote: post #9

Getting rid of ad supported content and SEO optimizations? Nothing of value will be lost. There is nothing “free” about the tracking and evilness of adtech. I have been paying $20/month for ChatGPT with web search just so I could avoid sites like yours. The first thing I see when I go to your site is a half page cookie banner that wants me to accept tracking.

But you still have to visit websites (or use apps) to do tasks online like booking tickets, getting stuff delivered, or paying bills. They are all tracking us. Do you also plan to pay for AI agents that do these chores on your behalf?
Post reply on HN