Live data from Hacker News

Blocking LLMs from your website cuts you off from next-generation search

johnjianwang.medium.com

41–50 of 82 posts

Re: Blocking LLMs from your website cuts you off from next-generation search

#41

Like everything else on the web, LLMs are going to eventually be ruined by marketing teams trying to get them to say "Pepsi" instead of "Coke."

Long live local LLMs!

Open models still have to get their data from somewhere, the only way they're any more immune is direct corruption. But marketers have shown time and time again that if there's any algorithmic crack in the wall, they will find it.

Re: Blocking LLMs from your website cuts you off from next-generation search

#42
post #6

This post gets the reason why people are cutting off LLMs exactly backwards and consequently completely fails to address the core issue. The whole reason people are blocking LLMs is precisely that they believe it kills the flow of readers to your content. The LLMs present your ideas and content, maybe with super-tiny attribution that nobody notices or uses [1], maybe with no attribution at all, and you get nothing. P…

Fair, if your content is your product, but I’m more than happy for every LLM on the planet to summarize my page and hype the virtues of my product to its user.

Re: Blocking LLMs from your website cuts you off from next-generation search

#43

Good, because this “next generation search” doesn’t cite sources, invents falsehoods, steals content, and doesn’t direct traffic to the site in question, which was the whole point of search engines in the first place . The fact LLM companies constantly keep getting dinged for ignoring every barrier we throw up to stop their scraping short of something like Anubis shows what their real goal is: theft, monopolization,…

It cites sources and while it generates less traffic, that traffic converts significantly better

Re: Blocking LLMs from your website cuts you off from next-generation search

#44

Good, because this “next generation search” doesn’t cite sources, invents falsehoods, steals content, and doesn’t direct traffic to the site in question, which was the whole point of search engines in the first place . The fact LLM companies constantly keep getting dinged for ignoring every barrier we throw up to stop their scraping short of something like Anubis shows what their real goal is: theft, monopolization,…

If your website is included in next gen search and the user asks for a source, then your website as a source will be included.

Re: Blocking LLMs from your website cuts you off from next-generation search

#45

Good, because this “next generation search” doesn’t cite sources, invents falsehoods, steals content, and doesn’t direct traffic to the site in question, which was the whole point of search engines in the first place . The fact LLM companies constantly keep getting dinged for ignoring every barrier we throw up to stop their scraping short of something like Anubis shows what their real goal is: theft, monopolization,…

If your website is included in next gen search and the user asks for a source, then your website as a source will be included.

Nobody clicks through, to a first approximation.

https://www.pewresearch.org/short-reads/2025/07/22/google-us...

Re: Blocking LLMs from your website cuts you off from next-generation search

#46
post #42
post #6

This post gets the reason why people are cutting off LLMs exactly backwards and consequently completely fails to address the core issue. The whole reason people are blocking LLMs is precisely that they believe it kills the flow of readers to your content. The LLMs present your ideas and content, maybe with super-tiny attribution that nobody notices or uses [1], maybe with no attribution at all, and you get nothing. P…

Fair, if your content is your product, but I’m more than happy for every LLM on the planet to summarize my page and hype the virtues of my product to its user.

Why do tech bros assume that every site is selling a product? There are blogs, personal web sites, communities, and open-source projects out there.

Re: Blocking LLMs from your website cuts you off from next-generation search

#48
post #37

I agree with the sentiment. I remember Gwern in an interview remarking something to the effect that if you make your writing and thoughts invisible to LLMs, then your thoughts are going to be invisible to the future, as LLMs are here to stay.

LLMs are not a replacement for the HTTP protocol, and people who want to see my site know the address.

Re: Blocking LLMs from your website cuts you off from next-generation search

#49
post #42

Earlier quoted context omitted.

Fair, if your content is your product, but I’m more than happy for every LLM on the planet to summarize my page and hype the virtues of my product to its user.

Why do tech bros assume that every site is selling a product? There are blogs, personal web sites, communities, and open-source projects out there.

If there's no product, and it's free, why would one care about it appearing in the output of an LLM? If it's so secret that it shouldn't, then perhaps it should be behind some auth anyway.
Post reply on HN