281–290 of 508 posts
Earlier quoted context omitted.
Block out IPv6 and see if that helps.
Why not block all odd v4 addresses while you're at it? I heard that that can reduce scraping volume by 50%!
It's easier and better to just block 0.0.0.0/1 half of the time, and 128.0.0.0/1 for the other half of the time. Switch every day at noon.
Bot traffic will be cut by 50%, and humans are all treated equally! It's a total win!
Earlier quoted context omitted.
I second this. My website exposes a cgit and 99% of the traffic now is AI scraping the sources, but the load is nowhere near DoS territory. And this is running on the cheapest VPS I could find. Not saying I'm not annoyed by the scraping; I am looking to block them, but I'm also not going to put the site behind the gatekeeper. If anything, Cloudflare must love AI scraping now for the same reason AV companies love malw…
Is there actually any plausible theory why "AI" would repeatedly scrape the same sites? Are there that many competing, completely independent AI labs? Is it cheaper to repeatedly scrape than to buffer the scraped data locally? (I find it very hard to imagine that it's easier to deal with changing/disappearing content than it is to stand up such a cache.)
"If they know you're spoofing, you're not spoofing hard enough." This stupid "war against bots" is going to lead to the downfall of the Internet and effectively turn it into another walled garden where only "approved" (anti-)user agents are allowed. Don't fall for the nonsense about "AI scrapers" --- it's just a way to manufacture consent.
Earlier quoted context omitted.
I can no longer access any website that's "protected" by Cloudflare. As soon a website enables that stuff… "Shoot, another one bites the dust." I wonder if the website owners realise at all how many actual users they lose by this sort of "protection."
I'm one of those who have enabled cloudflare on all of the sites I maintain. Additionally, Added turnstile on every form. I know some actual users get blocked. But the amount of spam we get without it, the amount of bot traffic simply overwhelming the server... It is just too much. Recently I also hard blocked all IPs from china Singapore India Pakistan Russia and whole of africa. Do I want to do it? No. But the amou…
At least for China, I imagine most of the real humans might use a VPN anyway
Earlier quoted context omitted.
That was more a (gloomy) outlook into the future, given Chrome's market dominance and tendency for unilateral actions in web standards.
I haven't ever noticed Cloudflare having any issues on Firefox, so presumably that implies any unilateral actions in web standards have been worked around by CF to provide the service to Firefox as well.
Earlier quoted context omitted.
I don't think regulation will stop web scraping, not least of which because it can be done from locations outside the jurisdiction of the regulations. > we have to acknowledge the system is broken The system is broken. It probably takes, what, 10 seconds or less to use a residential or foreign proxy, 6+ months to internationally track and prosecute a single offender? So like a million times more effort going the regu…
Just as criminal laws don't end all crimes, copyright laws and anti-scraping regulation won't end all scraping. But it will greatly reduce it and limit it to rogue actors. Two examples I can cite here are the laws against email spams and laws against unsolicited marketing calls - they had a definite impact in reducing both (even in India, from where I am, where implementation of laws are often lax).
However, a big difference with crimes involving the internet is that they can be launched from anywhere. In the real world, I can't steal from someone unless I'm physically present in the same country as my victim. On the internet, the US could outlaw scraping and Russia would keep doing it.
Earlier quoted context omitted.
I can no longer access any website that's "protected" by Cloudflare. As soon a website enables that stuff… "Shoot, another one bites the dust." I wonder if the website owners realise at all how many actual users they lose by this sort of "protection."
Cloudflare will just tell them that 70% traffic drop is because 70% of their traffic was bots, and everything is working fine, and hey, don't you want to upgrade to a paid plan to block 50% of the remainder? Think about how many bots will be blocked with that upgrade!
So no real benefit for bot detection here. Just a privacy nightmare for everyone else.
Internet Archive passed?