We all know wheret his is headed. In 5 years every useful content in the Web will be behind a paywall.
Served as word vomit.
AI crawlers need to be more respectful
71–80 of 128 posts
Re: AI crawlers need to be more respectful
#72[deleted]
Re: AI crawlers need to be more respectful
#73It would be nice to have more of a peer-to-peer infrastructure (torrent inspired) for serving big resources.
Re: AI crawlers need to be more respectful
#74Earlier quoted context omitted.
Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha. If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days. 73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.) Good luck =3
6 hours a minute? Are you joking? Click around a few times on any of your sites and looks like I'll be banned? Multi tasking? Opening multiple interesting links? Like what.
Generally for sites:
1. gets incrementally slower until firewall user rate limiting tokens refill the bucket (chokes >6MiB/min bandwidth use, and enforces abnormal traffic ban rules.)
2. Pauses serving a page if you spider though 6+ pages a minute (chokes speculative downloading)
3. if you violate site usage rules 4+ times in the past hour, than your get a 2 day IP ban
4. if you trip a spider trap, than you get a 5 day ban
5. If you are issued more than 5 context cookies, than the IP will get spammed with a captcha on every page for 5 days
6. If you violate any number of additional signatures (shodan etc.) than you get your IP block and route permanently banned. There is only 1 exception to this rule, and we don't share that with anyone.
7. The site content navigation is programmatically generated in JavaScript
8. The legal notice is very real for some people
Have a nice day friend, =)
Re: AI crawlers need to be more respectful
#75everything old is new again. i remember when someone at google started aggressively crawling del.icio.us from a desktop machine and i ended up blocking all employees...
Re: AI crawlers need to be more respectful
#76Earlier quoted context omitted.
It is 100% likely a cloud provider IP range. They are a persistent source of spam email servers, scrapers, and bot probes. The simple reason is the operators quickly dump a host, and the next user is left wondering why their legitimate site is instantly spelunking spam ban lists. It is the degenerative nature of cloud services... and unsurprisingly we end up often banning most parts of Digital Ocean, Amazon, Azure, G…
It's the degenerative nature of assuming an IP corresponds to a user. They have not corresponded to users for over a decade. I once discovered I'm banned on my mobile phone connection from at least one app which doesn't know that CGNAT exists (a very poor assumption for mobile phone apps in particular). If you must block IPs, do it as a last resort, make it based on some observable behavior, quickly instated when tha…
Really depends on the use-case, but yeah the response happens in a proportional manner.
We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs.
And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along with the active Tor exit nodes and known proxies lists (the hidden feature in context cookies).
Have a great day friend, =3
Re: AI crawlers need to be more respectful
#77"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.
10TB/day is, roughly, a single saturated 1Gbit link. In technical bandwidth terms, that is the square root of fuck all . The crazy thing here is that the target site is content to pay ~$700 for the volume of traffic that you can move through a single teeny-tiny included-at-no-extra-charge cat5 Ethernet link in one single day. And apparently, they're going to continue doing so.
Yeah that's an insane price for bandwidth, they need to move providers ASAP if that's the kind of fees they get.
Re: AI crawlers need to be more respectful
#78Shouldn't all sites have some kind of bandwidth / cost limiting in place? Not to say that AI crawlers shouldn't be more careful, but there are always malicious actors on the internet, seems foolish not to have some kind of defense in place
The big three cloud providers (AWS/GCP/Azure) have collectively decided that you don't want to set a spending limit actually, so they simply don't let you.
Isn't running a webserver on those kind of a silly idea for that reason?
Re: AI crawlers need to be more respectful
#79What amazes me that none of this is surprising, all this behavior (not just what's described in the post) is on par with what the companies are doing, and have been doing for decades... And yet there will be many people, including here on HN, who will just cheer these companies on because they spit out an "opensource model" or a 10-dollars-a-month subscription
Do you feel the same way about Google spidering for their commercial search engine?
Not originally since they just sent people to the site.
But these days where they just rip content from the site to give people as answers, completely depriving the site of traffic, yeah that seems basically just as bad as the AI bots.
Re: AI crawlers need to be more respectful
#80Earlier quoted context omitted.
Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha. If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days. 73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.) Good luck =3
> 73TB is a fair amount to have on a cloud From the article: "This was a bug in their crawler that was causing it to download the same files over and over again."
Must be a CDN choice issue, as most sites limit per IP daily file downloads, or have an account login with a quota limit.
People hitting the same files again sounds like a developer testing their code.