Blocking LLMs from your website cuts you off from next-generation search

Post by eric-burel » Sat, Aug 09, 2025, 9:09 AM UTC

skeledrew wrote: post #77

eric-burel wrote: post #75

Earlier quoted context omitted.

They are normal scrapers nothing specific to LLM as they are not yet used for training an LLM, unless I miss something from their architecture. So I don't get why they would be called LLM crawlers, when they are search engine crawlers. At least they could be called RAG crawlers for better nuance. The article linked in the post first sentence is more precise as it deals with scrapers: https://techcrunch.com/2025/08/04…

Crawling is done to discover and index content for search results (to relieve dependence on Google, etc). Scraping is done to get relevant content into the LLM's context window. And then the LLM generates the output. All the functions are there, so someone may emphasize just a subset to try making their point (which can cause issues if relevant context is left out, whether accidentally, ignorantly or maliciously). >…

Your answer is definitely clearer than the article, I get the point better, thanks for the feedback.

Post by kolinko » Wed, Aug 13, 2025, 7:00 PM UTC

sshine wrote: post #72

kolinko wrote: post #39

Earlier quoted context omitted.

> LLMs are not search engines, and I'm not gaining any followers or customers in any meaningful way because an LLM indexes my site. Friends of mine run a service company, and they already see a significant number of customers reach out because they found them using ChatGPT (et al), not Google. By significant I mean ~20% or so. Also, for e-commerce, Deep Research from OpenAI, is way better in doing product recommendat…

> Deep Research from OpenAI, is way better in doing product recommendations than Google Interesting, that’s not my experience and I’d be the first to replace Google if I could. I’ll have to try again.

Hm, example queries that failed you? Also, did you do DR from OpenAI, or others? (OpenAI's is a different league, and people sometimes confuse that with Perplexity's or Google's).

For me, the main place where it fails is specific links to the stores and specific prices / opportunities. But when I want to find an item that fits a need (e.g. "quietest mobile AC" or "best ultra short throw projector for my specific use case", "collagen supplement that has clinical confirmation of the quality...") it works way better than Google. And I tried many product categories.

Re: Blocking LLMs from your website cuts you off from next-generation search

Re: Blocking LLMs from your website cuts you off from next-generation search