Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI crawlers need to be more respectful

about.readthedocs.com

Post reply on HN
Print view View original

101–110 of 128 posts

Re: AI crawlers need to be more respectful

#101

Post by reaperman »

Joel_Mckay wrote: post #97
echoangle wrote: post #92

Earlier quoted context omitted.

Legally enforceable as in “you can block people who don’t follow the policy” or enforceable as in “you can sue them for money”?

If I recall it is considered theft-of-service if you bypass the posted site usage terms with an agent like a spider, and certainly a copyright violation for unauthorized content usage (especially in the context of a commercial venture.) One may be sued, but not because you parsed robots.txt wrong =3

> it is considered theft-of-service if you bypass the posted site usage terms

My understanding is that this is not accurate.

HiQ v LinkedIn established that this is only the case if you actually agreed to the terms of service. Such "agreement" only happens if the information is walled behind an account creation process, e.g. Facebook, Inc. v. Power Ventures, Inc. If it's just scraping publicly available webpages, the only legal issue with scraping would be unreasonably or obviously negligent scraping practices which lead to degradation or denial-of-service. And obviously the line for that would have to be determined in civil court.

eBay v. Bidder's Edge (2000) is the last case that I could find which even considered violation of robots.txt as very minor part of the judgement, but the findings were based far more on other things. Intel Corp. v. Hamidi also implicitly overruled the judgement in that that ruling (though not related to robots.txt, which was really just a very minor point in the first place).

Re: AI crawlers need to be more respectful

#102

Post by immibis »

Joel_Mckay wrote: post #57
DamonHD wrote: post #38

Earlier quoted context omitted.

When the bots come out of cloud services then the IPs are all over the place: it's much harder to do right these days.

Rate limiting firewalls and spider traps also work well... There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules. One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3

I mean, blocking random IP ranges is your prerogative if you don't want to have customers. The scrapers will find ways around, while actual users will be unable to use your site. Residential proxies are something like $5 for 1000.

Re: AI crawlers need to be more respectful

#103

Post by immibis »

bo1024 wrote: post #73

It would be nice to have more of a peer-to-peer infrastructure (torrent inspired) for serving big resources.

Never going to happen due to misaligned incentives. In 2024 everyone wants to keep their data behind lock and key. The commons is gone. Just look at the Google/Reddit deal.

Re: AI crawlers need to be more respectful

#104

Post by reaperman »

Joel_Mckay wrote: post #81
reaperman wrote: post #44

Earlier quoted context omitted.

Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”. Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.

The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)

Please review HiQ vs. LinkedIn - it hinged on the fact that HiQ hired crowdsourced workers (“turkers”) to create fake profiles through which to access LinkedIn’s platform (who had to agree to the ToS to create these accounts). The court found that hiQ expressly agreed to the user agreement when it created its corporate account on LinkedIn’s platform.

This doesn't apply if you don't ever agree to anything - which is the case if the information is not locked behind account creation.

Re: AI crawlers need to be more respectful

#105

Post by Joel_Mckay »

immibis wrote: post #102
Joel_Mckay wrote: post #57

Earlier quoted context omitted.

Rate limiting firewalls and spider traps also work well... There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules. One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3

I mean, blocking random IP ranges is your prerogative if you don't want to have customers. The scrapers will find ways around, while actual users will be unable to use your site. Residential proxies are something like $5 for 1000.

True, note domestic ISP IP ranges are published, and unless one deals internationally... don't bother serving people that will never buy anything from your firm anyways.

Domestic "Users" functioning as proxies will be tripping usage limits, and getting temporarily banned. Google does this by the way, try hammering their services and find out what happens.

Context cookies also immediately flag egregious multi-user routes, and if it is a ISP IP you know its a problem user. If it is over 15 users an hour per IP, than you can be 100% sure its a Tor proxy.

we ban over 243000+ IPs, and saw zero impact to our bottom line.

Have a nice day, =)

Re: AI crawlers need to be more respectful

#106

Post by Joel_Mckay »

immibis wrote: post #100
Joel_Mckay wrote: post #76

Earlier quoted context omitted.

Really depends on the use-case, but yeah the response happens in a proportional manner. We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs. And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along wit…

What I'm getting from this is that you hate having users almost as much as Reddit (which enshittified their website and banned all non-shit mobile apps and all search engines other than Google).

Imagine a world, where people walk into your business with a mask over their face saying horribly abusive things... while pretending they are your neighbors... And poof... they automatically vanish along with their garbage content.

They may visit again, but are less likely to mess with the platform. Note, cons never buy anything... ever... it is against their temperament.

I find it interesting several of cons on YC are upset by someone else's administrative policies. Reddit should enforce these policies too, or at least drop a country or pirate flag icon beside nasty posts...

Have a great day, and don't fear the ban hammer friend =3

Re: AI crawlers need to be more respectful

#107

Post by Joel_Mckay »

reaperman wrote: post #101
Joel_Mckay wrote: post #97

Earlier quoted context omitted.

If I recall it is considered theft-of-service if you bypass the posted site usage terms with an agent like a spider, and certainly a copyright violation for unauthorized content usage (especially in the context of a commercial venture.) One may be sued, but not because you parsed robots.txt wrong =3

> it is considered theft-of-service if you bypass the posted site usage terms My understanding is that this is not accurate. HiQ v LinkedIn established that this is only the case if you actually agreed to the terms of service. Such "agreement" only happens if the information is walled behind an account creation process, e.g. Facebook, Inc. v. Power Ventures, Inc. If it's just scraping publicly available webpages, the…

Hard to say, I seem to recall it was because some spider authors used session cookies to bypass the EULA (the page probe auto-clicks "I agree" to capture the session cookie), and faked user agent strings to spoof gogglebot to gain access to site content.

One thing is for certain, is its jurisdictional... and way too messy to be responsible for maintaining/hosting (the ambiguous copyright protection outside a research context looked way too risky.) =3

Re: AI crawlers need to be more respectful

#108

Post by Joel_Mckay »

reaperman wrote: post #104
Joel_Mckay wrote: post #81

Earlier quoted context omitted.

The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)

Please review HiQ vs. LinkedIn - it hinged on the fact that HiQ hired crowdsourced workers (“turkers”) to create fake profiles through which to access LinkedIn’s platform (who had to agree to the ToS to create these accounts). The court found that hiQ expressly agreed to the user agreement when it created its corporate account on LinkedIn’s platform. This doesn't apply if you don't ever agree to anything - which is t…

This gets complex fast, as a click-army is not necessarily violating the EULA.

However, if they scraped the content using these account credentials, than it becomes a problem in a commercial context.

If I recall, only journalists and academics could argue Fair use at that point.

Anyway, I didn't touch the project mostly for copyright and trademark risk concerns.

Have a great day =3

Re: AI crawlers need to be more respectful

#109

Post by EVa5I7bHFq9mnYK »

immibis wrote: post #102
Joel_Mckay wrote: post #57

Earlier quoted context omitted.

Rate limiting firewalls and spider traps also work well... There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules. One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3

I mean, blocking random IP ranges is your prerogative if you don't want to have customers. The scrapers will find ways around, while actual users will be unable to use your site. Residential proxies are something like $5 for 1000.

Exactly, I'm banned or captchaed from half of all web sites these days, because of the AI.

Re: AI crawlers need to be more respectful

#110

Post by troupo »

talldayo wrote: post #4
troupo wrote: post #2

What amazes me that none of this is surprising, all this behavior (not just what's described in the post) is on par with what the companies are doing, and have been doing for decades... And yet there will be many people, including here on HN, who will just cheer these companies on because they spit out an "opensource model" or a 10-dollars-a-month subscription

The situation didn't change when it was search index crawlers being called-out. At the end of the day, this sort of "abuse" is native to the world of the internet; like you said, it's decades old at this point. HN will cheer on a lot of things that are counter-intuitive to their wellbeing; open-weight models doesn't feel like one of them. You can't protest AI (or search engines) because after long enough people can't…

> HN will cheer on a lot of things that are counter-intuitive to their wellbeing; open-weight models doesn't feel like one of them.

Except for the fact that they come from undisclosed sources from a company that does this: https://x.com/Tantacrul/status/1794863603964891567

Post reply on HN