Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blocking LLMs from your website cuts you off from next-generation search

johnjianwang.medium.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Blocking LLMs from your website cuts you off from next-generation search

#61

Post by frozenseven »

debugnik wrote: post #28
frozenseven wrote: post #19

Why was this flagged? A difference in opinion is no excuse for censorship.

When I asked dang about why are very low quality submissions allowed despite them resulting in low quality discussion, he told me and I quote: > You can flag submissions that you think don't belong on Hacker News. Well, this is a very low quality submission in my eyes. A tiny read with an unsubstantiated, purely contrarian take that completely misses the point of the debate. Just to be clear, I think anyone is free t…

Well, this is a very high quality submission in my eyes. To the point and very much true (in my view). And, no, I'm not kidding or trolling. You have a strong opinion about AI, but so do other people. There's no debate to be had if you can just censor certain views out of existence on this website.

Re: Blocking LLMs from your website cuts you off from next-generation search

#62

Post by [deleted] »

jonplackett wrote: post #58
andrewmutz wrote: post #53

Earlier quoted context omitted.

Why would removing your content from LLM training data cause people to go and seek it out directly from you? Would removing your website from google search results cause people to go directly to your website?

This seems like a weird comparison - Google’s explicit purpose is to direct people to your site. An LLMs purpose is not.

[deleted]

Re: Blocking LLMs from your website cuts you off from next-generation search

#63

Post by piker »

PenguinCoder wrote: post #59
piker wrote: post #49

Earlier quoted context omitted.

If there's no product, and it's free, why would one care about it appearing in the output of an LLM? If it's so secret that it shouldn't, then perhaps it should be behind some auth anyway.

Not everything everyone does is for a profit motive. I'm not trying to sell you anything ; myself included, when you visit my site. It's just reading material.

Something being a product does not require a profit motive.

Re: Blocking LLMs from your website cuts you off from next-generation search

#64

Post by skeledrew »

jerf wrote: post #6

This post gets the reason why people are cutting off LLMs exactly backwards and consequently completely fails to address the core issue. The whole reason people are blocking LLMs is precisely that they believe it kills the flow of readers to your content. The LLMs present your ideas and content, maybe with super-tiny attribution that nobody notices or uses [1], maybe with no attribution at all, and you get nothing. P…

Users don't seek content for the attribution; that's extra noise, unless there's reason to contact the attributed. And given that many websites offer an inefficient flow to content, made of ads and/or unnecessarily animated things for example, the LLM is merely improving the experience for the user.

Re: Blocking LLMs from your website cuts you off from next-generation search

#65

Post by skeledrew »

jonplackett wrote: post #58
andrewmutz wrote: post #53

Earlier quoted context omitted.

Why would removing your content from LLM training data cause people to go and seek it out directly from you? Would removing your website from google search results cause people to go directly to your website?

This seems like a weird comparison - Google’s explicit purpose is to direct people to your site. An LLMs purpose is not.

The point being made is that just as the search engine was the primary means for users to discover content yesterday, so the LLM agent will become the primary means tomorrow. And that content doesn't have to be in the training data, but if an agent is unable to access some particular content, then it won't be discovered by users. Similar to if a rewatch engine is unable to access it.

Re: Blocking LLMs from your website cuts you off from next-generation search

#67

Post by johnnienaked »

skeledrew wrote: post #64
jerf wrote: post #6

This post gets the reason why people are cutting off LLMs exactly backwards and consequently completely fails to address the core issue. The whole reason people are blocking LLMs is precisely that they believe it kills the flow of readers to your content. The LLMs present your ideas and content, maybe with super-tiny attribution that nobody notices or uses [1], maybe with no attribution at all, and you get nothing. P…

Users don't seek content for the attribution; that's extra noise, unless there's reason to contact the attributed. And given that many websites offer an inefficient flow to content, made of ads and/or unnecessarily animated things for example, the LLM is merely improving the experience for the user .

"Why look at a sunset when you can read a summary about one from an AI?"

-Someone, somewhere, eventually

Re: Blocking LLMs from your website cuts you off from next-generation search

#68

Post by skeledrew »

eric-burel wrote: post #32

The first sentence of the article is literally wrong as it conflates LLM and the search part of a RAG (retrieval augmented generation, when you mix a web search and an LLM). Blocking bots cuts you off from the next-generation search, because it cuts you off from search at all. So far, blocking LLM simply prevents you from being part of the training dataset, which is not the same thing. Please stop upvoting such bad c…

> blocking LLM simply prevents you from being part of the training dataset

That's narrow. Perplexity, and other LLM agent services, do perform a regular web search to gain context, before generation their output. How else would they have access to recent data when the underlying LLM's knowledge cutoff is usual at least a few weeks?

Re: Blocking LLMs from your website cuts you off from next-generation search

#69

Post by jerf »

skeledrew wrote: post #64
jerf wrote: post #6

This post gets the reason why people are cutting off LLMs exactly backwards and consequently completely fails to address the core issue. The whole reason people are blocking LLMs is precisely that they believe it kills the flow of readers to your content. The LLMs present your ideas and content, maybe with super-tiny attribution that nobody notices or uses [1], maybe with no attribution at all, and you get nothing. P…

Users don't seek content for the attribution; that's extra noise, unless there's reason to contact the attributed. And given that many websites offer an inefficient flow to content, made of ads and/or unnecessarily animated things for example, the LLM is merely improving the experience for the user .

The whole conversation here is about incentives for the content creators, not the user.

Yes, as a user I'd like everything served to me on a silver platter, for free, on demand, and completely and 100% aligned with my interests exclusively with no thought given to anybody else... but that's not a realistic world. In the real world, if the content providers have no reason to provide content, they won't.

I kind of hate the connotations of "content provider", that neutral term that implies that it is all "content" that can just be measured in megabytes or something, but I mean the full richness here of the term, individual producers, small businesses, big business, everybody. Even my personal site, if I'm not getting something out of it, however intangible it may be, I wouldn't do it. I'd be mighty pissed if I lose a job someday because I get accused of just spewing out LLM content that the LLM can only spew out because of my own original ideas/formulation of ideas being on the internet.

Re: Blocking LLMs from your website cuts you off from next-generation search

#70

Post by jerf »

piker wrote: post #42
jerf wrote: post #6

This post gets the reason why people are cutting off LLMs exactly backwards and consequently completely fails to address the core issue. The whole reason people are blocking LLMs is precisely that they believe it kills the flow of readers to your content. The LLMs present your ideas and content, maybe with super-tiny attribution that nobody notices or uses [1], maybe with no attribution at all, and you get nothing. P…

Fair, if your content is your product, but I’m more than happy for every LLM on the planet to summarize my page and hype the virtues of my product to its user.

Enjoy the brief window of LLMs "hyping the virtues of your product" to its users for free. In 2030 that's not going to sound realistic at all. And I feel I'm being generous pushing it up to 2030, the first "sponsored training data" either already exists or will probably be out this year, the only question being whether it will be publicly admitted to or not.
Post reply on HN