As someone who writes web scrapers for a living, I have only come across one site where I have been unable to reliably extract the information we need. If we were more flexible, we would be able to deal with this site too. Defending yourself from scrapers is an arms race you are almost certain to loose.
It is not possible to detect and block Chrome headless
111–120 of 174 posts
Re: It is not possible to detect and block Chrome headless
#112If you want to detect if a human is visiting your site, open an ad popup with a big close button directly over the content. A human being will always, 100% of the time, immediately close the popup. Automation won't care.
OK, but that is guaranteed to annoy users. Plus, I think you’re underestimating the intelligence of the people writing scrapers — obviously they’re going to visit the site manually and see what appears to be a fingerprinting measure. Then they’ll update the scraper to close that pop up. There are no effective solutions to this problem.
Re: It is not possible to detect and block Chrome headless
#113[deleted]
Re: It is not possible to detect and block Chrome headless
#114As someone who writes web scrapers for a living, I have only come across one site where I have been unable to reliably extract the information we need. If we were more flexible, we would be able to deal with this site too. Defending yourself from scrapers is an arms race you are almost certain to loose.
What site?
I’d guess LinkedIn or Facebook.
I’ve had to make a lot of fake accounts to get just a decent amount of data from them.
Re: It is not possible to detect and block Chrome headless
#115Earlier quoted context omitted.
In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…
> Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Honest question, would this open the site up to legal liability? You can never identify a bot 100%, so if you intentionally provided false information to an otherwise legitimate user, and that user is harmed by your false information, isn't that a bre…
if you intentionally provided false information to an otherwise legitimate user, and that user is harmed by your false information, isn't that a breach of contract that would make you liable for damages based on the harm your bad data caused?
What contract?
Re: It is not possible to detect and block Chrome headless
#116Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?
Or just recompile Chromium with a few changes ;)
Re: It is not possible to detect and block Chrome headless
#117Earlier quoted context omitted.
> Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Honest question, would this open the site up to legal liability? You can never identify a bot 100%, so if you intentionally provided false information to an otherwise legitimate user, and that user is harmed by your false information, isn't that a bre…
if you intentionally provided false information to an otherwise legitimate user, and that user is harmed by your false information, isn't that a breach of contract that would make you liable for damages based on the harm your bad data caused? What contract?
Terms of service. It is not considered a contract if you are just an anonymous user, but once you take an affirmative action like creating an account you're explicitly agreeing to that and they are legally binding. The parent comment did not specify anonymous and plenty of bots create accounts to scrape with.
But even aside from the explicit contractual terms, even without one, you cannot just run around acting in bad faith, right? If you put up a site called AccuratePrices.com and then for some users you knowingly and intentionally provide false information, isn't that something like fraud?
Re: It is not possible to detect and block Chrome headless
#118Earlier quoted context omitted.
If it's a video site, I expect the video to play when I land, e.g. youtube. I'm initiating on purpose by browsing
Regarding YouTube in particular, I tend to open up videos in background tabs for later viewing and find it very annoying that they start playing automatically before I get around to that tab. I did go there to watch the video—eventually. Just not the second that the page finishes loading. YMMV. A persistent setting to enable or disable auto-play would be ideal.
Chrome doesn't autoplay videos in background tabs until you focus them.
Re: It is not possible to detect and block Chrome headless
#119As someone who writes web scrapers for a living, I have only come across one site where I have been unable to reliably extract the information we need. If we were more flexible, we would be able to deal with this site too. Defending yourself from scrapers is an arms race you are almost certain to loose.
thats why i usually block all the ip range from hosting providers.
Re: It is not possible to detect and block Chrome headless
#120Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…
Give it time my friend: they don't need to detect headless browsers, they just need to restrict your DOM-given freedom altogether.
Some prick is bound to make some fancy non-DOM web framework using web assembly and turn the internet into a DRM-ridden mess.