Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

website-auditor.io

Post reply on HN
Print view View original

11–20 of 66 posts

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#11

Post by chollida1 »

ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

> It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end.

I mean, its not. Lots of LLM's focused on finance do this already.

Bloomberg's own ASKB produces results and provides links back to the source documents or urls that it references so people who care about correctness can verify the results.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#12

Post by jefftk »

ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

You can do it the same way human does: recall a fact from memory, and then search to identify a citation. The citation isn't "here's why I think this" but instead "here's where you can verify this".

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#13

Post by amelius »

ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

Then the least they can do is cite ALL their sources. At least somewhere on their website.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#16

Post by WarmWash »

ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

This is why it's dumb when inferencing with a model to ask "...and cite all sources". Total waste of time and likely to make the response worse.

BUT

On models with web search, they can scan the sources in context, and those are pretty good at correct citations. However it's still a "trust, but verify" situation.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#17

Post by pfdietz »

SecretDreams wrote: post #6
ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

Lol, this same behavior comes up more often than you'd expect in academia too. Many many citations that nobody cross checks that also don't support the originally cited piece of information.

It's hilariously common online as well. Bullshitter makes a claim, someone calls them on the BS, they put up a link, and (lo and behold) the link when examined does nothing to defend the claim. Often the link directly contradicts the claim.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#18

Post by SecretDreams »

pfdietz wrote: post #17
SecretDreams wrote: post #6

Earlier quoted context omitted.

Lol, this same behavior comes up more often than you'd expect in academia too. Many many citations that nobody cross checks that also don't support the originally cited piece of information.

It's hilariously common online as well. Bullshitter makes a claim, someone calls them on the BS, they put up a link, and (lo and behold) the link when examined does nothing to defend the claim. Often the link directly contradicts the claim.

Yep..the online link posters hit close to home. Big reason I hardly even engage in discourse anymore. Got tired of verifying links that don't have the claimed content in them.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#20

Post by brookst »

This is a weird complaint.

Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia.

There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site?

Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most sites deserve links because they were crawled seems like a statistical fallacy. You could say the same thing about the percent of sites crawled by Google versus ever showing up on first page of results.

Post reply on HN