Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

website-auditor.io

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#21

Post by DrDeese »

brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

I wouldn't even consider it a complaint. For a builder, I would consider it an opportunity...

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#22

Post by passwordoops »

eterm wrote: post #5

Are we saying that it's now a problem that we're not getting scraped? This appears to be a new generation of "SEO", marking itself as a service for getting into AI results? This is not the future I want to be a part of. Perhaps it's inevitable that after a break from everything being driven by money that LLMs will now be ruined by people spending $X to get into AI to make back $X+1, leading to an arms race of ever in…

LLM's were obviously going be ruined by two factors:

1- LLM Crawler Optimization: the new SEO

2- Weightings-For-Pay: for a fee have your product or service come up more frequently in associated answers

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#23

Post by sparkling »

Blocking known bot identifiers via robots.txt does nothing by the way. Too many labs are running sneaky crawlers that do not respect robots.txt. You will need to take extreme measures: blocking basically all datacenter IP ranges, VPN IPs, aggressive rate limiting, etc.

Blocking LLM crawlers has become the number one use case for our IP database customers at https://focsec.com/

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#24

Post by ordersofmag »

ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

99% of AI users these days aren't using a 'raw' LLM. They are interacting with a harness that includes tools to do search of the live (or recently crawled) web. And so the output users actually see could absolutely include correct citation of sources. So the 'architecturally impossible' bit may be technically correct but it is not practically relevant. Now whether those harnesses do a good job of orchestrating LLM output to get accurate citations (and whether they are transparent about their process) is another thing entirely. But if you're contemplating 'what LLM's can do' and aren't taking into account the harness and tooling ecosystem they are embedded in then you're missing the point.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#26

Post by mark_l_watson »

Odd results for me. Last month I tested five AI chat sites, with web search turned off, and four of them had a shadow of information about me as a person (what kind of books I write, what tech I use, and a random bit of other information). The linked site gave me a zero score because it was testing if the AI models recommended my site for business or sales queries.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#27

Post by kyleblarson »

brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

The question is very important. What if you ask/tell an AI "I need a 2 bedroom vacation rental in Park City for a trip this fall". If you only get Airbnb results you are missing a lot of the true answer.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#28

Post by ssl-3 »

ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

> It is perfectly common to find that a "source" does not contain the statements.

It is commonly this way right now, but it doesn't have to stay this way forever.

It is also common, in my usage at least, to iteratively brow-beat the bot into paring its statements down to those that which are supportable by its sources. Doing so just takes repetition, and that repetition takes time and burns more tokens.

With the present state of things, the prompts to get moving on this and to guide the ultimate response into something that is verifiably supportable by outside sources can usually be simple and largely generic.

They're easy enough prompts that a subagent can produce them.

(I've done it myself with Codex subagents and it worked very well, aside from the unsustainable burn rate that did not fit my budget.)

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#29

Post by SpikeyCoder »

brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

If the goal of the web was just to transmit objective facts like 'who created Linux', this wouldn't be a problem at all. Wikipedia handles that pretty dece.

The issue arises when we move away from objective trivia and into subjective, localized buying intent, which is where the web monetizes itself.

If I ask an LLM 'who created Linux?', there is one right answer. But if I ask an LLM 'who are the best commercial roofers in Houston?', there isn't one right answer. There are dozens of highly qualified local businesses that do possess unique value, unique pricing, and unique availability.

When an LLM answers that roofing question, it typically cites 3 to 5 businesses. The other 40 legitimate roofing companies in the area are left out. My study isn't arguing that every single one of those 40 companies deserves to be in the answer; it's pointing out that those 40 companies currently have no idea they are being left out.

In the Google era, if you weren't on page 1, you could look at Search Console, see your ranking, check your backlinks, and understand why. In the AI Search era, businesses are being scraped to build these answers, but they have zero telemetry on whether they are actually making the cut. This index is just an attempt to provide that missing telemetry.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#30

Post by tiffanyh »

brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

I just asked ChatGPT-5.6 that question, no source was given.

Not even Wikipedia.

Post reply on HN