Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

website-auditor.io

Post reply on HN
Print view View original

61–66 of 66 posts

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#61

Post by deadbabe »

phoghed wrote: post #53
deadbabe wrote: post #45

This is why we need the HTTP 402 standard to become common. If websites charge pennies per AI crawl, they will make more money than ever being reference in that 6.2% of websites that get cited (of which even another small percent get any follow through that leads to a sale or ad click) HTTP 402 also basically extends the pay per token model people have gotten used to with AI model providers, except applied to the who…

Seems like the opposite if the problem you’re trying to solve is “my site isn’t cited over someone else’s”. HTTP 467 - pls cite me, I’ll pay you.

Do you want to be cited, or do you want to make money?

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#62

Post by deadbabe »

ndriscoll wrote: post #57
deadbabe wrote: post #45

This is why we need the HTTP 402 standard to become common. If websites charge pennies per AI crawl, they will make more money than ever being reference in that 6.2% of websites that get cited (of which even another small percent get any follow through that leads to a sale or ad click) HTTP 402 also basically extends the pay per token model people have gotten used to with AI model providers, except applied to the who…

The problems with micropayments are 1. The market for lemons. In fact sites trying to "monetize their content" are the most likely to be lemons, so just asking is a signal that your "content" is not worth anything. 2. If your information does have value, it competes in a market with other sites full of high quality information that aren't trying to monetize it, meaning your specific information has to be specifically…

Yes but imagine a platform instead where watching any YouTube video is paid for by a micro transaction. Transparently.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#63

Post by ndriscoll »

deadbabe wrote: post #62
ndriscoll wrote: post #57

Earlier quoted context omitted.

The problems with micropayments are 1. The market for lemons. In fact sites trying to "monetize their content" are the most likely to be lemons, so just asking is a signal that your "content" is not worth anything. 2. If your information does have value, it competes in a market with other sites full of high quality information that aren't trying to monetize it, meaning your specific information has to be specifically…

Yes but imagine a platform instead where watching any YouTube video is paid for by a micro transaction. Transparently.

In a sense, that already exists. e.g. spotify or youtube premium. In practice only extremely exceptional individuals are going to see anything from it because creative markets are oversaturated, it's trivial to enter them, and people have finite time, so almost everyone is just not worth anyone's attention, let alone compensation.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#65

Post by phoghed »

deadbabe wrote: post #61
phoghed wrote: post #53

Earlier quoted context omitted.

Seems like the opposite if the problem you’re trying to solve is “my site isn’t cited over someone else’s”. HTTP 467 - pls cite me, I’ll pay you.

Do you want to be cited, or do you want to make money?

It depends. If I’m Danon and I’m paying for SEO to drive traffic to my recipes page it’s because I want to sell yogurt. I’d rather have OpenAI drive traffic to my site than give me a few cents for my recipes themselves to be in their training corpus.

Re: Only 8.9% of sites block AI crawlers, but 94.8% are never cited in AI answers

#66

Post by JohnTHaller »

It's also basically impossible to block most AI crawlers. Most completely ignore robots.txt. Some claim to respect it but I've found the majority that claim to also ignore it. You have to block their useragent and then their IPs if they spoof it and use botnets like some China-based AI crawlers.
Post reply on HN