Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

website-auditor.io

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#41

Post by knollimar »

jefftk wrote: post #12
ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

You can do it the same way human does: recall a fact from memory, and then search to identify a citation. The citation isn't "here's why I think this" but instead "here's where you can verify this".

I don't understand why verifying isn't in the loop. If you gave an LLM: "is this statement consistent with this RAG data?" it would do well.

It feels lazy that this isn't built into the harness in some adversarial citation review checkbox.

Also the questions LLMs ask are so inhuman. It's like it's some contrarian trying to win an argument over getting an answer.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#42

Post by altmanaltman »

SpikeyCoder wrote: post #29
brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

If the goal of the web was just to transmit objective facts like 'who created Linux', this wouldn't be a problem at all. Wikipedia handles that pretty dece. The issue arises when we move away from objective trivia and into subjective, localized buying intent, which is where the web monetizes itself. If I ask an LLM 'who created Linux?', there is one right answer. But if I ask an LLM 'who are the best commercial roofe…

Not really, ahrefs has already started working on this and there are several tools out there that track GEO.

Also what is the incentive for me to rank on LLMs? People do not use it to click sources or even take action. Do you have any proof that being ranked as the best roofer in houston drives sales more than not being ranked as it? If not, why should i care as the roofer?

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#43

Post by llm_nerd »

brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

It isn't a complaint. It's an ad. This is literally an ad for some sort of "get cited by AI" service.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#44

Post by razodactyl »

ButlerianJihad wrote: post #3

It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…

Are you sure? Because this sounds like GPT3-era understandings of LLM-isms

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#45

Post by deadbabe »

This is why we need the HTTP 402 standard to become common.

If websites charge pennies per AI crawl, they will make more money than ever being reference in that 6.2% of websites that get cited (of which even another small percent get any follow through that leads to a sale or ad click)

HTTP 402 also basically extends the pay per token model people have gotten used to with AI model providers, except applied to the whole web, with the added privacy benefit in that there is no need for sellers of content to “know their customer”, and indeed it may even be impossible to do so because of how the payment gateways operate.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#46

Post by SpikeyCoder »

altmanaltman wrote: post #38
brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

Its a marketing post for a tool that offers visibility to site owners. Honestly these type of "neutral data reports" masquerades should be flagged or adequately disclosed.

I disclosed on my first comment how I got my data. I thought it was an interesting find, and one that really hinders the non-technical small business owner from figuring out how to ensure their company finds their way to the right customer.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#47

Post by martinald »

SpikeyCoder wrote: post #29
brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

If the goal of the web was just to transmit objective facts like 'who created Linux', this wouldn't be a problem at all. Wikipedia handles that pretty dece. The issue arises when we move away from objective trivia and into subjective, localized buying intent, which is where the web monetizes itself. If I ask an LLM 'who created Linux?', there is one right answer. But if I ask an LLM 'who are the best commercial roofe…

That's not _entirely_ true. Search Console now has a Generative AI page where you can see impressions per page now. Bing has something similar.

Also, I would assume that really "GEO" is just like "SEO". If you rank on the '1st page' of results for whatever common searches, you are _very_ likely to rank the same way for LLM questions, because all the LLM is doing (nearly all of the time for 'best roofers in Houston') is doing a web search and summarising the first x results. So if you are on page 1 for that term, it's very likely IME that you will get mentioned on LLM answers for that.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#48

Post by amelius »

brookst wrote: post #37
amelius wrote: post #13

Earlier quoted context omitted.

Then the least they can do is cite ALL their sources. At least somewhere on their website.

So an HTML page with a list of essentially every page on every site on the internet? What do you think, maybe a trillion URLs?

Yes. Whatever makes their lawyers nervous.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#49

Post by phoghed »

SpikeyCoder wrote: post #29
brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

If the goal of the web was just to transmit objective facts like 'who created Linux', this wouldn't be a problem at all. Wikipedia handles that pretty dece. The issue arises when we move away from objective trivia and into subjective, localized buying intent, which is where the web monetizes itself. If I ask an LLM 'who created Linux?', there is one right answer. But if I ask an LLM 'who are the best commercial roofe…

Good, let’s hope there’s no SEO arms race for this too. Maybe the best roofers have a chance of getting cited over the ones with the best SEO agency. Seems unlikely though.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#50

Post by netcan »

brookst wrote: post #20

This is a weird complaint. Let’s say I ask “who created Linux?” Claude correctly tells me Linus Torvalds, and links to Wikipedia. There are probably thousands, maybe hundreds of thousands of other sites that have that some piece of information. Are LLMs supposed to link to every site? Most sites do not have unique information at all, and even sites that do rarely contain only unique information. Implying that most si…

This is not a complaint.

I think your assessment of this result being as expected... but this is about the LLM equivalent of SEO becoming an area of interest

Post reply on HN