91–100 of 128 posts
Sites need to start suing crawler operators for bandwidth costs.
Earlier quoted context omitted.
Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”. Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.
The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)
Earlier quoted context omitted.
This doesn't solve their bandwidth costs which is their real problem with these bots.
My 100 mbps upload bandwidth at home is free (apart from the monthly 35€ payment). Useless bots will get stuck downloading from me instead of hogging readthedocs.
1. I intentionally made sure my crawler was slow (I prefer batch processing workflows in general, and this also has the effect of not needing a machine gun crawler rate)
2. For data updates, I made sure to first do a HEAD request and only access the page if it has actually been changed. This is good for me (lower cost), the site owner, and the internet as a whole (minimizes redundant data transfer volume)
Regarding individual site policies, I feel there’s often a “tragedy of the commons” dilemma for any market segment subject to aggregator dominance:
- individual sites often aggressively hide things like pricing information and explicitly disallow crawlers from accessing them
- humans end up having to access them: this results in a given site either not being included at all, or accessed once but never reaccessed, causing aggregator data to go stale
- aggregators often outrank individual sites due to better SEO and likely human preference of aggregators, because it saves them research time
- this results in the original site being put at a competitive disadvantage in SEO, since the their product ends up not being listed, or listed with outdated/incorrect information
- that sequence of events leads to negative business outcomes, especially for smaller businesses who often already have a higher chance of failure
Therefore, I believe it’s important to have some sort of standard policy that is implemented and enforced at various levels: CDNs, ISPs, etc.
The policy should be carefully balanced to consider all these factors as well as having a baked in mechanism for low friction amendment based on future emergent effects.
This would result in a much better internet, one that has the property of GINI regulation, ensuring well-distributed outcomes that are optimized for global socioeconomic prosperity as a whole.
Curious to hear others’ perspectives about this idea and how one would even kick off such an ambitious effort.
Earlier quoted context omitted.
Do you feel the same way about Google spidering for their commercial search engine?
I don't. Just 3 AI spiders put more load on our servers than all search engine spiders and all human traffic combined. Some numbers I have handy from before I blocked the bots: ClaudeBot drove more requests through our Redmine in a month than it saw in the combined 5 years prior to ClaudeBot. Bytespider accounted for 59% of the total traffic to our Git server. Amazonbot accounted for 21% of the total traffic to our G…
Not just AI: here is my current side-quest: https://www.earth.org.uk/RSS-efficiency.html Over 99% of the bandwidth (and CPU) taken by the biggest podcast / music services simply on polling feeds is completely unnecessary. But ofc pointing this out to them gets some sort of "oh this is normal, we don't care" response because they are big enough to know that eg podcasters need them.
I don't disagree with your post. But: RSS downloads are at an all time low, and that's a bad thing.
They're at an all time low because Spotify and Apple both fetch feeds from centralized servers. 1000 subscribers no longer means 24000ish daily feed fetches, it means 48. With keep alive or H2, these services simply don't reconnect. The number of IPs that hit me from Apple, for instance, is probably only double digits.
Since Apple and Spotify both sit between me and the listeners, they eliminate the privacy that listeners would otherwise enjoy. It also forces podcasters to go to them to find out how many people are subscribed, which means lots of big databases instead of one database that I host for my customers.
Centralization of feed checking carries huge risks, in my opinion, especially as both Apple and Spotify make moves to also become the hosting providers.
Earlier quoted context omitted.
The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)
Legally enforceable as in “you can block people who don’t follow the policy” or enforceable as in “you can sue them for money”?
One may be sued, but not because you parsed robots.txt wrong =3
Not only that, even commoncrawl had issues (about a year ago) where AWS couldn't keep up with the demand for downloading the WARCs. As someone who written a lot of crawling infrastructure and managed large scale crawling operations, respectful crawling is important. That being said it always seems like google has had a massively unfair advantage for crawling not only with budget but with brandname, and perceived valu…
VCs and other startup culture evangelists are always challenging founders to figure out what their ‘unfair advantage’ is.
That’s the name of the game.
The words "AI" and "respectful" don't belong in the same sentence. The mere concept of generative AI is disrespectful.
Earlier quoted context omitted.
It's the degenerative nature of assuming an IP corresponds to a user. They have not corresponded to users for over a decade. I once discovered I'm banned on my mobile phone connection from at least one app which doesn't know that CGNAT exists (a very poor assumption for mobile phone apps in particular). If you must block IPs, do it as a last resort, make it based on some observable behavior, quickly instated when tha…
Really depends on the use-case, but yeah the response happens in a proportional manner. We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs. And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along wit…