Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
1–10 of 66 posts
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#2[flagged]
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#3It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a legal or medical opinion supported by research.
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#4It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…
For standard responses you're not wrong, but increasingly there are a lot of LLMS that are essentially doing RAG against search results. For example this is I believe how Kagi works, and Google AI overviews. I have increasingly seen Claude and chat GPT also doing the same thing where instead of answering a question from the knowledge Bank they will do a web search and cite the responses that were used.
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#5Are we saying that it's now a problem that we're not getting scraped?
This appears to be a new generation of "SEO", marking itself as a service for getting into AI results?
This is not the future I want to be a part of.
Perhaps it's inevitable that after a break from everything being driven by money that LLMs will now be ruined by people spending $X to get into AI to make back $X+1, leading to an arms race of ever increasing X, to the detriment of users.
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#6It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…
Lol, this same behavior comes up more often than you'd expect in academia too. Many many citations that nobody cross checks that also don't support the originally cited piece of information.
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#7[deleted]
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#8Are we saying that it's now a problem that we're not getting scraped? This appears to be a new generation of "SEO", marking itself as a service for getting into AI results? This is not the future I want to be a part of. Perhaps it's inevitable that after a break from everything being driven by money that LLMs will now be ruined by people spending $X to get into AI to make back $X+1, leading to an arms race of ever in…
I get that pushback. My goal with this index/report wasn't to write a new 'AI SEO' playbook or encourage people to start gaming the system. The goal was simply to shine a bit of light on what's going on. Right now, there is a massive information asymmetry: AI companies are turning their assistants into primary search engines, but webmasters have zero visibility into whether their sites are actually being cited in those live answers.
You're absolutely right to be concerned about an arms race. If the data showed that doing X, Y, and Z guaranteed a citation, we'd be right back to the worst days of keyword stuffing.
But what this data actually shows is the opposite: even if you do everything 'right' (allow the retrieval bots, provide perfect machine-readable schema, don't block anything), you still have a 94.8% chance of never being cited. How LLMs cite is entirely opaque.
I built this to give site owners a baseline measurement of what is actually happening to their content today (as a free, anonymous view), so they can make an informed decision on whether keeping their doors open to these crawlers is actually worth it.
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#9Makes me wonder if paid Medium and paid news are input to AI training. Surely, they are.
Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers
#10It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end. Every link they're giving you to support their statements is tacked on because it may vaguely match the tokens it just generated. It is perfectly common to find that a "source" does not contain the statements. You cannot expect an LLM to write you a Wikipedia article, much less a l…
This is true, but good LLMs (even local, consumer-sized ones these days) can do this much better than … whatever it is that Google’s AI overviews use.
Think about your coding harness: the model reads a bunch of files into the context, and it generally doesn’t forget/hallucinate which lines came from which file.