61–70 of 508 posts
Earlier quoted context omitted.
Probably everyone on HN singing their praises for the past 10 years.
And my og comment getting downvoted on this very intellectual forum that definitely isn't an echo chamber
Cloudflare is known to use fingerprinting to detect scrapers For example, they use JA3 fingerprints and match them against the UA to block stuff like cURL while allowing OkHttp (Android clients) - but this can be easily be spoofed with packages such as CycleTLS [1]. I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecolog…
...in the age of AI, does anyone have an actual solution for keeping out bots while preserving the privacy of humans? Obviously this is terrible, but I think there's a possibility it's the least terrible option? Another option is IP reputation, which I think is worse. Or scanning a code with a non-rooted phone, which I think is even worse than that!
The only solution is regulation . If all content created by anyone has a copyright, how does an implicit opt-in (which is what happens if you don't create a robots.txt file for your website) for scraping make any sense? Moreover, even if you have a robots.txt, AI (or whatever) bots often don't respect it (or use workarounds - they outsource scraping of such "restricted" sites to unethical third-parties to get the dat…
As for issues like bots overloading websites or using too many resources scaling laws will take care of it quickly, it’s not like you can’t serve thousands of RPS from a Raspberry Pi these days.
Cloudflare is known to use fingerprinting to detect scrapers For example, they use JA3 fingerprints and match them against the UA to block stuff like cURL while allowing OkHttp (Android clients) - but this can be easily be spoofed with packages such as CycleTLS [1]. I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecolog…
it's all for nothing, because Cloudflare's scraping protection works about as well as a $5 padlock - good enough to dissuade bored teens, not good enough to dissuade even an amateur burglar. if someone wants to scrap your publicly visible data, they will. there's nothing you can do.
So if you need to prevent bot abuse, but also don't want an ugly captcha every time someone goes to sign up, is there a better option?
The tool "Anubis" uses proof of work instead
But in principle I agree that there's no good answer to this, scraping _is_ useful and I bet most of us here had scraped something, it is AI company and their use of human's material for training without consent and return that led us to this (I know botting exists in forum since forum is a thing but it is easily solved by human moderators and keyword filter)
Also by default addons.mozilla.org is a privileged site so of course they include google tracking in it and they get the proper fingerprint no matter what you have configured.
Earlier quoted context omitted.
> keeping out bot You can forget about it. It is not possible. Simple as that.
Let's say I'm selling concert tickets. How do I prevent bots from buying up all the tickets and scalping them?
Earlier quoted context omitted.
How does proof of work stop bots?
Because it destroys the economics of scraping. It’s too expensive with proof of work, or at least not as economically viable
Earlier quoted context omitted.
This is why I have two separate browsers. If you want to do official stuff like paying for things you need to get through cloudflare.
Firefox added profile switching recently. Works good. (That said, I still keep separate machines. One for doing "official" things, the other for everything else)
does it? same binary, same machine, same display, same 781 other heuristics.