Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Content Independence Day: no AI crawl without compensation

blog.cloudflare.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Content Independence Day: no AI crawl without compensation

#31

Post by grg0 »

quixoticaxolotl wrote: post #28
grg0 wrote: post #25

Earlier quoted context omitted.

That is not what I understood, and it sounds terrible. What if you're not a crawler but random Joe surfing the internet? Clearly Joe should see content without payment? So they need some way to tell the crawler and Joe apart, and presumably they require the crawler to set certain request headers. The headers aren't just to issue the payment, it's to identify the crawler in the first place?

Joe will be fine. Cloudflare is pretty good at differentiating humans from bot traffic - see how we do it here: https://developers.cloudflare.com/turnstile/ The idea behind the headers is to allow bots to bypass automatic bot filtering, not blockade all regular traffic. In other words: - we block bots (the website owner can configure how aggressively we block) - unless they say they're from an AI crawler we've vetted…

Ok, well, thanks for the clarification.

Re: Content Independence Day: no AI crawl without compensation

#32

Post by DamonHD »

OutOfHere wrote: post #29
techjamie wrote: post #26

Earlier quoted context omitted.

Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.

If we are going for cheaper, how is it cheaper than an HTTP 429 error? It's not.

Virtually nothing pays attention to 429s that I have observed. More things pay attention to 500s and 503s. Some however use those as a trigger to repoll immediately.

Re: Content Independence Day: no AI crawl without compensation

#33

Post by xg15 »

quixoticaxolotl wrote: post #28
grg0 wrote: post #25

Earlier quoted context omitted.

That is not what I understood, and it sounds terrible. What if you're not a crawler but random Joe surfing the internet? Clearly Joe should see content without payment? So they need some way to tell the crawler and Joe apart, and presumably they require the crawler to set certain request headers. The headers aren't just to issue the payment, it's to identify the crawler in the first place?

Joe will be fine. Cloudflare is pretty good at differentiating humans from bot traffic - see how we do it here: https://developers.cloudflare.com/turnstile/ The idea behind the headers is to allow bots to bypass automatic bot filtering, not blockade all regular traffic. In other words: - we block bots (the website owner can configure how aggressively we block) - unless they say they're from an AI crawler we've vetted…

Thanks for replying! Do you have some provision for false positives as well, like sending a captcha in the body of the 402 response? (So in case the client was a human and not a bot, they could still try to solve the captcha)

Re: Content Independence Day: no AI crawl without compensation

#34

Post by Retric »

Mars008 wrote: post #30
Retric wrote: post #17

Earlier quoted context omitted.

Serving poisoned text can be so cheap it’s effectively free as long as you don’t give them a lot of links.

Yeh, and say goodby to google search. You didn't want to be there anyway, right?

Google makes it easy to identify their bot. Often people want to do this to give them more access.

People care about AI companies because they’re ignoring robots.txt etc.

Re: Content Independence Day: no AI crawl without compensation

#35

Post by OutOfHere »

Retric wrote: post #17
OutOfHere wrote: post #8

Earlier quoted context omitted.

Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances. Definitely don't rewrite your web server more efficiently in Rust instead. /s

Serving poisoned text can be so cheap it’s effectively free as long as you don’t give them a lot of links.

Another thing that doesn't make sense is why it has to be poisoned text. Why can't it just be a mix of whitespace? I doubt anyone is using LLMs with streaming inputs to determine whether to continue reading the page.

Re: Content Independence Day: no AI crawl without compensation

#36

Post by Retric »

OutOfHere wrote: post #35
Retric wrote: post #17

Earlier quoted context omitted.

Serving poisoned text can be so cheap it’s effectively free as long as you don’t give them a lot of links.

Another thing that doesn't make sense is why it has to be poisoned text. Why can't it just be a mix of whitespace? I doubt anyone is using LLMs with streaming inputs to determine whether to continue reading the page.

Company’s actively harming you should be discouraged, preferably by running them out of business. Whitespace doesn’t do that and makes it easy to identify when the crawlers fail.

Swapping meaning poisons the LLM but makes it really difficult for a preprocessing step to understand the difference between good and bad inputs.

Re: Content Independence Day: no AI crawl without compensation

#37

Post by felurx »

OutOfHere wrote: post #29
techjamie wrote: post #26

Earlier quoted context omitted.

Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.

If we are going for cheaper, how is it cheaper than an HTTP 429 error? It's not.

I suppose they're going for a trade-off between "cheaper" and "satisfying" (as in, the satiafaction of sticking it to unethical shitty companies)

Re: Content Independence Day: no AI crawl without compensation

#38

Post by felurx »

trhway wrote: post #23
mhuffman wrote: post #16

So are they going to try and IP gate them or trust that AI companies that literally stole the info they used to make the base models will now respect robots.txt entries?

Every one likes net neutrality when the one is benefitting from it, yet the one immediately jumps at the opportunity to break net neutrality on their services if it allows to increase profit by price discrimination (which may take a shape of extracting a rent from some subset of consumers like seems to be in this case) .

AI scrapers aren't a subset of the users/readers/audience of a website, they are parasites to society that harm their prey by causing huge costs.

Re: Content Independence Day: no AI crawl without compensation

#39

Post by apoorvagni0 »

felurx wrote: post #38
trhway wrote: post #23

Earlier quoted context omitted.

Every one likes net neutrality when the one is benefitting from it, yet the one immediately jumps at the opportunity to break net neutrality on their services if it allows to increase profit by price discrimination (which may take a shape of extracting a rent from some subset of consumers like seems to be in this case) .

AI scrapers aren't a subset of the users/readers/audience of a website, they are parasites to society that harm their prey by causing huge costs.

why? Aren't the AI scrapers working under a company which creates value therefore exists in the first place?
Post reply on HN