61–70 of 169 posts
There is no technical mechanism whereby it is actually possible to allow people to read your webpage and not use it for other things. You can’t give responses that say “this is ok for indexing but not for training”. Anyone trying to sell you this sort of technology is lying.
Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…
Not sure why Anubis is getting so much hype on HN, but honestly, it is not the solution. A real solution would use behavioral modeling. Most browser fingerprinting issues are already largely solved anyway.
Earlier quoted context omitted.
> And if you're getting a turnstile challenge it's unclear how it's different than an anubis challenge. You are guaranteed to pass an Anubis challenge eventually [0], whereas it's possible to get stuck forever in an infinitely-looping Turnstile challenge. > If you're outright blocked, it's probably a site decision (eg. block all VPNs or block everyone not from a given country) rather than cloudflare's. Cloudflare blo…
>You are guaranteed to pass an Anubis challenge eventually [0] That's a double edged sword because bots will eventually get through too, and unlike humans, their time is dirt cheap. >Cloudflare blocks legitimate users itself sometimes [1]. I never ran into this issue despite using seemingly maximally suspicious configs like tor browser. I can't say the same for some other vendors.
On the contrary I only want to visit a few pages on a given site and have an entire laptop at my disposal. Meanwhile for bots efficiency is key. A serious scraper (ie the type of actor that actually causes material problems for site operators) is performing tens or hundreds of pages loads per second per core spread across thousands of sites. Making a single page load take even half a second of cpu time is a massive win for the site operator.
Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…
PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.
Earlier quoted context omitted.
PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.
You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.
Earlier quoted context omitted.
Google's web scraping functionality has been acting as a ddos for more than two decades. I've seen literally hundreds of reports of them attacking websites and taking them down, where there's nothing you can do but accept the traffic, or get delisted This is unfortunately nothing new. There's no correct way to tell them to fuck off, they do not care, and they never will do. People have even taken them to court over t…
If a site cannot handle traffic from the real Googlebot that is a serious issue with the site itself since it's actually pretty conservative Also I should note there are lots of fake Googlebots...
Please consider installing one of the many PoW schemes such as anubis rather than use these cloudflare "features". I increasingly encounter outright blocks rather than any sort of captcha when visiting cloudflare "protected" sites. Each individual site isn't particularly important to me but it's depressing to watch the process unfold like this. You really are choosing to erode the core basis of the internet if you go…
> Please consider installing one of the many PoW schemes such as anubis Why not go all the way and mine monero instead of just completely wasting the work?
Earlier quoted context omitted.
How are they playing both sides? I thought their scraping products were also about having it behave and not take down systems
I mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests. This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled wi…
Earlier quoted context omitted.
PoW schemes like Anubis don't work. Increasingly bots are using headless browsers and are basically able to solve captchas, proof-of-work(s) and basically bypass all any any attempts to block them. It's becoming impossible to stop bots from hammering your sites/services for unwanted traffic.
You are asserting that they don't work without explanation or evidence. Meanwhile it isn't clear why they wouldn't and indeed they appear to accomplish the stated goal of severely rate limiting scrapers.
Oh, no, that isn't how it works — you made a claim first (albeit indirectly), with no evidence. It's on you