311–320 of 508 posts
Earlier quoted context omitted.
> Cloudflare's scraping protection works about as well as a $5 padlock It sure seems to keep me, the casual visitor, far away from just about any site they "protect". I have zero desire to alter my browsing configuration or use extra tools to get around turnstile, I'd rather not even visit the site in the first place.
>, I'd rather not even visit the site in the first place Until your bank, airline, and tax ministry start using them.
Earlier quoted context omitted.
I think people get annoyed when it's suggested they spend time optimising or even re-writing their websites to handle high traffic loads just to cater to AI bots ripping their content. It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good ou…
Curious, but how do the bots figure out the combinations? Or do you have links to the diffs from other sites? I assume the diff takes two files in query parameters or something.
I did try removing some of the links without success. I guess once they have them they just keep checking.
Earlier quoted context omitted.
> but unless you do PoW (which is also ecologically a nightmare) Can you expand? I don't see a problem with some napkin math. 5W load for 2 seconds is 0.002Wh (we have to let smartphones pass and not by doing PoW for 10s of seconds). 8 billion checks a day for a year = 8GWh.
I stand corrected. It's not a nightmare scenario (as for Bitcoins) - but I'm still of the idea that "useless" computations should be avoided (as we should avoid having 10MB websites). In any case, according to some napkin math done by Kimi 2.6 (which by itself is probably already consuming more than all of my PoW challenges for the upcoming 5 years) - the situation looks incredibly in favor of PoW: https://www.kimi.c…
If some computation is "useless" but it serves it's purpose, it's not useless.
The reason why bitcoin network expends so much energy is down to tokenomics, not the system of PoW itself. At equilibrium we expect the power usage to be (blocks/hr) x (BTC/block) x ($/BTC) x (kWh/$), so it's a function of the BTC price and emission rate.
PoW in other context has way different driving factors. In this case, the marginal improvement of fetching the site again for AI bots isn't enough to cover the PoW cost. The PoW cost is outweighed by the net bandwidth cost of all the parties.
Is there a deal between Google and Cloudflare to make non-Chrome browsers harder to use? The pressure to use Chrome keeps increasing, and the amount of ad filtering you can do in Chrome keeps decreasing.
Earlier quoted context omitted.
Stack Overflow was outside of the Cloudflare network for years , and anti-abuse was maybe 3 or 4 full-time jobs – much of which still needs to be done, because Cloudflare's anti-bot protection hasn't actually stopped it. Most UGC sites are not as big as Stack Overflow was at its peak.
errr... so anything related to UGC now has a lower bound of 3-4 FTE? Sure, I'll hire a team of content moderators next time I think about putting a comment form under my blog...
Earlier quoted context omitted.
I think people get annoyed when it's suggested they spend time optimising or even re-writing their websites to handle high traffic loads just to cater to AI bots ripping their content. It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good ou…
What I do is have more strict rate limits for non logged in users. You tell them to log in if they hit the rate limit. For non logged in users, you have a rate limit not just for IP, but also for /24 and /16. Forget about IPv6, IPv4 scarcity is a feature not a bug.
Similar to the one SQLite had: https://www2.sqlite.org/forum/forumpost/7d3eb059f81ff694?t=h
Each IP only makes ~1 request though so easy to detect after the fact.
I guess they will run out of IPs at some point so maybe if I had logged each one forever and shown a challenge only to them, it would have fixed it eventually. Just depends how big their pool of IPs is.
Earlier quoted context omitted.
The goal of Cloudflare’s fingerprinting is to detect whether a user agent appears to be a legitimate human. It’s not to identify human users across websites.
That is not a good excuse for requiring overly complicated and overly specific software.
Earlier quoted context omitted.
> we all know what happens if you just open your site up without these, especially with AI bots which hammer servers and are in effect a legalized DDoS system No, we don't know. I honestly do not understand the problem. I run websites, both static and non-static. Granted, my sites aren't exactly the most popular internet go-to destinations, but I should be seeing this DDoS too, right? I do see lots of requests. Nothi…
You get downvoted for these opinions but I agree. Most people that complain that their servers get hammered by AI bots are those that run very unoptimized servers that can only handle like 100 rps. I've never had any issues with any of my moderately optimized websites. A $10 VPS can handle sooo much traffic.
Cloudflare is known to use fingerprinting to detect scrapers For example, they use JA3 fingerprints and match them against the UA to block stuff like cURL while allowing OkHttp (Android clients) - but this can be easily be spoofed with packages such as CycleTLS [1]. I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecolog…
JA3 fingerprinting is really not a serious deterrent, there are many ways to get around that. curl-impersonate works. You can even just use an actual Chrome instance with the devtools protocol, seems to pass as long as you don't use headless mode. The WebGL fingerprinting thing is cute, too. I guess it'll buy them some time since off-the-shelf solutions are going to probably not handle this well yet. That said, as lo…
By virtue of incompetent and ignorant Devs and middle managers. Our by virtue of greed and maliciousness.
Yeah yeah never attribute to malice what can be explained by stupidity... This time no. It's both.
Earlier quoted context omitted.
> but unless you do PoW (which is also ecologically a nightmare) Can you expand? I don't see a problem with some napkin math. 5W load for 2 seconds is 0.002Wh (we have to let smartphones pass and not by doing PoW for 10s of seconds). 8 billion checks a day for a year = 8GWh.
I stand corrected. It's not a nightmare scenario (as for Bitcoins) - but I'm still of the idea that "useless" computations should be avoided (as we should avoid having 10MB websites). In any case, according to some napkin math done by Kimi 2.6 (which by itself is probably already consuming more than all of my PoW challenges for the upcoming 5 years) - the situation looks incredibly in favor of PoW: https://www.kimi.c…