31–40 of 128 posts
Shouldn't all sites have some kind of bandwidth / cost limiting in place? Not to say that AI crawlers shouldn't be more careful, but there are always malicious actors on the internet, seems foolish not to have some kind of defense in place
While the crawling is disrespectful, it seems RTD could find a cheaper host for their files. At my work we have a 10G business fiber line and serve >1PB per month for around $1,500. Takes 90% of the load off our cloud services. Took me just a couple weeks to set up everything.
Just 2 buggy crawlers seems not that many, sure they each had large impact, but given that there are likely hundreds if not thousands of such crawlers out there it's a rather small number. It seems that most crawlers are actually respectful.
How many foundation AI companies are there? 2 is a pretty big chunk of that pie.
"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.
Over 99% of the bandwidth (and CPU) taken by the biggest podcast / music services simply on polling feeds is completely unnecessary. But ofc pointing this out to them gets some sort of "oh this is normal, we don't care" response because they are big enough to know that eg podcasters need them.
"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.
If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days.
73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.)
Good luck =3
Earlier quoted context omitted.
Do you feel the same way about Google spidering for their commercial search engine?
I don't. Just 3 AI spiders put more load on our servers than all search engine spiders and all human traffic combined. Some numbers I have handy from before I blocked the bots: ClaudeBot drove more requests through our Redmine in a month than it saw in the combined 5 years prior to ClaudeBot. Bytespider accounted for 59% of the total traffic to our Git server. Amazonbot accounted for 21% of the total traffic to our G…
"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.
Try page rate-limiting (6 hits a minute is plenty for a human), and then pop up a captcha. If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days. 73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.) Good luck =3
"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.
Respect to them for not naming names, that's the classy move, but I wouldn't blame them if they did.