21–30 of 128 posts
Just 2 buggy crawlers seems not that many, sure they each had large impact, but given that there are likely hundreds if not thousands of such crawlers out there it's a rather small number. It seems that most crawlers are actually respectful.
How many foundation AI companies are there? 2 is a pretty big chunk of that pie.
"One crawler downloaded 73 TB of zipped HTML files in May 2024, with almost 10 TB in a single day. This cost us over $5,000 in bandwidth charges, and we had to block the crawler." Wow. That's some seriously disrespectful crawling.
Shouldn't all sites have some kind of bandwidth / cost limiting in place? Not to say that AI crawlers shouldn't be more careful, but there are always malicious actors on the internet, seems foolish not to have some kind of defense in place
Just 2 buggy crawlers seems not that many, sure they each had large impact, but given that there are likely hundreds if not thousands of such crawlers out there it's a rather small number. It seems that most crawlers are actually respectful.
The only web crawler that did anything for me was Google, as Google sent an appreciable amount of traffic. Referrers from Bing were almost undetectable: the joke among my black hat SEO friends at the time was that you could rank for money keywords like "buy wow gold" and get 10 hits. Then there were the Chinese crawlers like Baidu that would crawl at 10x the rate of Google but send zero referrers. And then there were crawlers looking for copyrighted images that cost me money to accommodate even if they never sent me cease and desist letters.
As much as I hate the Google monopoly I couldn't afford having my site crawled like that without any benefit to me.
It's an awful situation for the long term though because it prevents new entrants. Right now I am thinking about a new search engine for a vertical where a huge number of products are available from different vendors and when you do find results from Google they are sold out at least 70% of the time. I hate to think it's going to get harder to make something.
Earlier quoted context omitted.
How many foundation AI companies are there? 2 is a pretty big chunk of that pie.
I think there are a thousand wannabe companies all trying to suck up as much data as they can; not a sustainable situation in any way.
You would be fooling yourselves if you think such a firm cared about robots.txt or page tags.
We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of another jurisdiction their domestic firm wouldn't be liable for theft of service or copyright infringement.
It was my understanding AI/ML does not change legal obligations in business, but the firm probably found someone to build that dubious project eventually...
Spider traps and rate-limiting are good options too. =3
What amazes me that none of this is surprising, all this behavior (not just what's described in the post) is on par with what the companies are doing, and have been doing for decades... And yet there will be many people, including here on HN, who will just cheer these companies on because they spit out an "opensource model" or a 10-dollars-a-month subscription
Do you feel the same way about Google spidering for their commercial search engine?
Just 3 AI spiders put more load on our servers than all search engine spiders and all human traffic combined.
Some numbers I have handy from before I blocked the bots:
ClaudeBot drove more requests through our Redmine in a month than it saw in the combined 5 years prior to ClaudeBot.
Bytespider accounted for 59% of the total traffic to our Git server.
Amazonbot accounted for 21% of the total traffic to our Git server.
Google has never even been close to breaking out of the single-digit-percentages of any metric.
While the crawling is disrespectful, it seems RTD could find a cheaper host for their files. At my work we have a 10G business fiber line and serve >1PB per month for around $1,500. Takes 90% of the load off our cloud services. Took me just a couple weeks to set up everything.