Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Content Independence Day: no AI crawl without compensation

blog.cloudflare.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Content Independence Day: no AI crawl without compensation

#21

Post by PaulDavisThe1st »

TekMol wrote: post #9

Currently, what I do is that when an IP requests insane amounts of URLs on my server (especially when its all broken urls causing 404s) I look up the IP and then block the whole organization. For example today some bot from the range 14.224.0.0-14.255.255.255 got crazy and caused a storm of 404s. Dozens per second for hours on end. So I blocked the range like this: iptables -A INPUT -m iprange --src-range 14.224.0.0-…

At git.ardour.org, we block any attempt to retrieve a specific commit. Trying to do so triggers fail2ban putting the IP into blocked status for 24hrs. They also get a 404 response.

We wouldn't mind if bots simply cloned the repo every week or something. But instead they crawl through the entire reflog. Fucking stupid behavior, and one that has cost us an extra $50/month even with just the 404.

Re: Content Independence Day: no AI crawl without compensation

#22

Post by GGO »

TekMol wrote: post #9

Currently, what I do is that when an IP requests insane amounts of URLs on my server (especially when its all broken urls causing 404s) I look up the IP and then block the whole organization. For example today some bot from the range 14.224.0.0-14.255.255.255 got crazy and caused a storm of 404s. Dozens per second for hours on end. So I blocked the range like this: iptables -A INPUT -m iprange --src-range 14.224.0.0-…

I like rate-limiting. I know none of my users will need more than 10qps. I set that for all routes, and all bots get throttled. I can also have much higher rate-limit for authenticated users. Have not had bots slamming me - they just get 429s

Re: Content Independence Day: no AI crawl without compensation

#23

Post by trhway »

mhuffman wrote: post #16

So are they going to try and IP gate them or trust that AI companies that literally stole the info they used to make the base models will now respect robots.txt entries?

Every one likes net neutrality when the one is benefitting from it, yet the one immediately jumps at the opportunity to break net neutrality on their services if it allows to increase profit by price discrimination (which may take a shape of extracting a rent from some subset of consumers like seems to be in this case) .

Re: Content Independence Day: no AI crawl without compensation

#24

Post by cryptonector »

grg0 wrote: post #10
yablak wrote: post #7

Earlier quoted context omitted.

Details here: https://blog.cloudflare.com/introducing-pay-per-crawl/

This requires good faith on behalf of the crawler? So it's DOA; why even bother implementing this? Also, what a piece of zero-trust shit the web is becoming thanks to a couple of shit heads who really need to extract monetary value out of everything. Even if this non-solution were to work, the prospect of putting every website behind Cloudsnare is not a good one anyway. What the web needs right now, to be honest, is…

It's Cloudflare. That means they are good at DoS and DDoS protection. AI crawlers are basically DoS agents. I think CF can start with an honor system that also has attached to it the implied threat to block crawlers from all CF hosted content, and that is a pretty big hammer to hit the abusers with.

So I'm cautiously optimistic. Well, I suppose pessimistic too: if this works what this will mean is that all contents will end up moving into big player hosting like CF.

Re: Content Independence Day: no AI crawl without compensation

#25

Post by grg0 »

xg15 wrote: post #13
grg0 wrote: post #10

Earlier quoted context omitted.

This requires good faith on behalf of the crawler? So it's DOA; why even bother implementing this? Also, what a piece of zero-trust shit the web is becoming thanks to a couple of shit heads who really need to extract monetary value out of everything. Even if this non-solution were to work, the prospect of putting every website behind Cloudsnare is not a good one anyway. What the web needs right now, to be honest, is…

Maybe I didn't understand the proposal completely yet, but wouldn't the crawler only have to cooperate (send the right headers, implement that auth framework, etc) if they want to pay? The standard response to a crawler is a 402 Payment Required response, probably as a result of an aggressive bot detection. So essentially, it's turning a site's entire content into an API: Either sign up for an API key or get blocked.…

That is not what I understood, and it sounds terrible. What if you're not a crawler but random Joe surfing the internet? Clearly Joe should see content without payment? So they need some way to tell the crawler and Joe apart, and presumably they require the crawler to set certain request headers. The headers aren't just to issue the payment, it's to identify the crawler in the first place?

Re: Content Independence Day: no AI crawl without compensation

#26

Post by techjamie »

OutOfHere wrote: post #8
agentultra wrote: post #4

A nice attempt and another layer for the swiss cheese of technology it will take to try and ease the burden AI companies are putting on people trying to run websites. I'd be cautious about relying on just the good will of Cloudflare. It's unfortunate that we need honeypots and tarpits to trap AI scrapers just so that our hosting bills don't get hosed. It's taking a good chunk of value out of running a site on the Int…

Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances. Definitely don't rewrite your web server more efficiently in Rust instead. /s

Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.

Re: Content Independence Day: no AI crawl without compensation

#27

Post by xg15 »

grg0 wrote: post #25
xg15 wrote: post #13

Earlier quoted context omitted.

Maybe I didn't understand the proposal completely yet, but wouldn't the crawler only have to cooperate (send the right headers, implement that auth framework, etc) if they want to pay? The standard response to a crawler is a 402 Payment Required response, probably as a result of an aggressive bot detection. So essentially, it's turning a site's entire content into an API: Either sign up for an API key or get blocked.…

That is not what I understood, and it sounds terrible. What if you're not a crawler but random Joe surfing the internet? Clearly Joe should see content without payment? So they need some way to tell the crawler and Joe apart, and presumably they require the crawler to set certain request headers. The headers aren't just to issue the payment, it's to identify the crawler in the first place?

The writeup doesn't talk about actively misbehaving crawlers a lot, but this bit implies for me that the headers are for the "happy path", I.e. crawlers wanting to pay:

> Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing.

I don't see how it would make sense otherwise, as the requirements for crawlers include applying for a registration with Cloudflare.

Who in their right mind would jump through registration hoops only so they can not access a site? This wouldn't even keep away the crawlers that are operating today.

I agree there has to be some way to distinguish crawlers from regular users, but the only way I can see how this could be done is with bot detection algorithms.

...which are imperfect and will likely flag some legitimate human users as bots. So yes, this will probably leading to web browsing becoming even more unpleasant.

Re: Content Independence Day: no AI crawl without compensation

#28

Post by quixoticaxolotl »

grg0 wrote: post #25
xg15 wrote: post #13

Earlier quoted context omitted.

Maybe I didn't understand the proposal completely yet, but wouldn't the crawler only have to cooperate (send the right headers, implement that auth framework, etc) if they want to pay? The standard response to a crawler is a 402 Payment Required response, probably as a result of an aggressive bot detection. So essentially, it's turning a site's entire content into an API: Either sign up for an API key or get blocked.…

That is not what I understood, and it sounds terrible. What if you're not a crawler but random Joe surfing the internet? Clearly Joe should see content without payment? So they need some way to tell the crawler and Joe apart, and presumably they require the crawler to set certain request headers. The headers aren't just to issue the payment, it's to identify the crawler in the first place?

Joe will be fine. Cloudflare is pretty good at differentiating humans from bot traffic - see how we do it here: https://developers.cloudflare.com/turnstile/

The idea behind the headers is to allow bots to bypass automatic bot filtering, not blockade all regular traffic. In other words:

- we block bots (the website owner can configure how aggressively we block) - unless they say they're from an AI crawler we've vetted, as attested by the signature headers - in which case we let them pay - and then they get to access the content

(Disclosure: I wrote the web bot auth implementation Cloudflare uses for pay per crawl)

Re: Content Independence Day: no AI crawl without compensation

#29

Post by OutOfHere »

techjamie wrote: post #26
OutOfHere wrote: post #8

Earlier quoted context omitted.

Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances. Definitely don't rewrite your web server more efficiently in Rust instead. /s

Many of these tarpits deliberately serve the data at an excruciatingly low speed to ease the burden on the server resources. It's cheaper than quickly serving the same crawlers your entire website at max speed constantly.

If we are going for cheaper, how is it cheaper than an HTTP 429 error? It's not.

Re: Content Independence Day: no AI crawl without compensation

#30

Post by Mars008 »

Retric wrote: post #17
OutOfHere wrote: post #8

Earlier quoted context omitted.

Feel free to waste your expensive outgoing bandwidth running malware. It is a genius idea really from the cloud companies to enrich their balances. Definitely don't rewrite your web server more efficiently in Rust instead. /s

Serving poisoned text can be so cheap it’s effectively free as long as you don’t give them a lot of links.

Yeh, and say goodby to google search. You didn't want to be there anyway, right?
Post reply on HN