Live data from Hacker News

Blocking LLMs from your website cuts you off from next-generation search

johnjianwang.medium.com

31–40 of 82 posts

Re: Blocking LLMs from your website cuts you off from next-generation search

#31
This article is based on the false assumption that use of a site by an LLM directs any user traffic to it whatsoever.

Why would anyone choose to anonymously and freely provide content to LLMs? Actually the only use case for that is deliberately seeding misinformation. Which is likely already happening and will soon be the majority of the content accessible to LLMs regardless of what blocking legitimate content providers choose to use.

Re: Blocking LLMs from your website cuts you off from next-generation search

#32
The first sentence of the article is literally wrong as it conflates LLM and the search part of a RAG (retrieval augmented generation, when you mix a web search and an LLM). Blocking bots cuts you off from the next-generation search, because it cuts you off from search at all. So far, blocking LLM simply prevents you from being part of the training dataset, which is not the same thing. Please stop upvoting such bad content it really makes Hackernews a terrible place for staying informed on LLMs.

Re: Blocking LLMs from your website cuts you off from next-generation search

#33
Once again I see someone mistaking an LLM regurgitating (in a right, wrong, misleading way, who knows) your content for people "accessing" your content. If the LLM sits between me and my reader and acts as a filter (because the information is rehashed, because maybe sometimes doesn't reference me), basically my goal is to provide information to tools for other companies to make money? I don't write for money, but if you remove also the basic human interaction reader-writer, I might as well really write in my notes.

Re: Blocking LLMs from your website cuts you off from next-generation search

#34

> how many of you wouldn’t hook up your website to Google? If there was a paid-only search engine with dubious ethics practices that was overwhelming my site with traffic in order resell search trained off of (among other things) my personally generated content, I would absolute block it. LLMs are not search engines, and I'm not gaining any followers or customers in any meaningful way because an LLM indexes my site.…

FWIW most of the inbound traffic to startups websites that I know enough to ask about is coming from ChatGPT.

Re: Blocking LLMs from your website cuts you off from next-generation search

#36

Like everything else on the web, LLMs are going to eventually be ruined by marketing teams trying to get them to say "Pepsi" instead of "Coke."

I don't know what you mean ... by including "eventually" in that sentence.

Re: Blocking LLMs from your website cuts you off from next-generation search

#38
This article is stupid.

You write content so that you get paid, usually through ads and clicks. If people aren't seeing your content because am LLM has consumed it and is regurgitating it and taking your ad clicks, then there's no benefit for you only for the LLM. You're doing the work of Sam Altman and helping him attain his multibillionaire status and you get nothing in return.

Re: Blocking LLMs from your website cuts you off from next-generation search

#39

> how many of you wouldn’t hook up your website to Google? If there was a paid-only search engine with dubious ethics practices that was overwhelming my site with traffic in order resell search trained off of (among other things) my personally generated content, I would absolute block it. LLMs are not search engines, and I'm not gaining any followers or customers in any meaningful way because an LLM indexes my site.…

> LLMs are not search engines, and I'm not gaining any followers or customers in any meaningful way because an LLM indexes my site.

Friends of mine run a service company, and they already see a significant number of customers reach out because they found them using ChatGPT (et al), not Google. By significant I mean ~20% or so.

Also, for e-commerce, Deep Research from OpenAI, is way better in doing product recommendations than Google. That's my goto place to find most stuff novadays (e.g. I purchased dancing shoes, pants, air cleaners, an air conditioner, supplements and a ton of other things using the recommendations of DR - no search engine comes even close to it)

Re: Blocking LLMs from your website cuts you off from next-generation search

#40
Good, because this “next generation search” doesn’t cite sources, invents falsehoods, steals content, and doesn’t direct traffic to the site in question, which was the whole point of search engines in the first place.

The fact LLM companies constantly keep getting dinged for ignoring every barrier we throw up to stop their scraping short of something like Anubis shows what their real goal is: theft, monopolization, and reality authoring.

Post reply on HN