Cool, I think the new captchas use mouse entropy that would be an interesting test since remote usually go straight to the pixel point.
Most touchscreens go straight to the pixel point too.
It is not possible to detect and block Chrome headless
101–110 of 174 posts
Re: It is not possible to detect and block Chrome headless
#102Earlier quoted context omitted.
Video should never play unless actively initiated by the user. That would fix the metrics, as the headless browser probably wouldn't initiate the video playback
If it's a video site, I expect the video to play when I land, e.g. youtube. I'm initiating on purpose by browsing
Yeah, I definitely don't. But I think a persistent "auto-play" toggle (default "off") is ok.
Re: It is not possible to detect and block Chrome headless
#103Earlier quoted context omitted.
Video should never play unless actively initiated by the user. That would fix the metrics, as the headless browser probably wouldn't initiate the video playback
If it's a video site, I expect the video to play when I land, e.g. youtube. I'm initiating on purpose by browsing
The first thing I do when hitting a YouTube URL is stop the video. Then I'll either run youtube-dl on the URL, or just paste it straight into a proper video player (VLC).
Re: It is not possible to detect and block Chrome headless
#104If you want to detect if a human is visiting your site, open an ad popup with a big close button directly over the content.
A human being will always, 100% of the time, immediately close the popup. Automation won't care.
Re: It is not possible to detect and block Chrome headless
#105Earlier quoted context omitted.
> When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique. For one thing, detecting that a scraper is a scraper is the problem, not the prelude to the problem. You might as well block them at that point, if you feel you can reliably dete…
Choosing a believable alteration is just as important as detecting a browser. For price scrapping even 3% changes to prices can do a lot of damage and is much harder to detect than completely bogus data.
That's a fair point, and it's conceivable this could work. But if the scrapers have had a legitimate source of truth in your data for a meaningful amount of time before you pull the trigger on this, it will get significantly harder.
In practice I'm talking about data which is very multidimensional and has more nuanced metrics than something like a price point. As an example, let's say a certain company exposes sequentially crawlable online orders without authentication. A scraper is going to reason about the data instead of just accepting it, and it's going to look for relatively minute errors. In particular, if it suddenly pulls in many more orders in a single day, or if a particular product shoots up in popularity, or if the price of an item in inventory suddenly changes from what it was historically, an alert will fire off.
As a real world example, I was working on a project to scrape all orders for a publicly traded fast food company that offered online orders and delivery. In the middle of one quarter, without warning, we suddenly saw activity that looked legitimate, but noticeably different order data occurring in a way that significantly increase revenue projections. My colleagues and I basically didn't trust the data whatsoever until we could find a localized (and not well publicized online) promotion that accounted for the changes.
Basically, I'm saying that it can be done, but it's hard and not a silver bullet. If the data is being used as part of a timeseries, then data meaningful enough to disrupt operations if going to be noticed and manually reviewed to identify a narrative that explains it; if the data is believably false but not meaningful enough to change a trend, it's not really going to matter to them anyway.
Re: It is not possible to detect and block Chrome headless
#106Earlier quoted context omitted.
All of which is defeated by OCR.
Good point. OCR powered web scraping is even available out of the box nowadays. https://a9t9.com/kantu/docs/scraping#ocr
It is not the OCR that is costly. It is the JavaScript execution to render the page so you can do the OCR. You can even increase the JavaScript execution cost if suspicious.
You will also have to automate all page variations and the traditional challenges (login, captcha, user behavior fingerprinting, ...)
At the end the development time, cost and server cost will kick you out of business if you are too dependent on the information or you start to loose money every time you scrap.
Re: It is not possible to detect and block Chrome headless
#107Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?
There's a similar "analog hole" for video DRM, too.
I wonder how long it will be before someone comes up with the idea of using iPhone style facial recognition to tell whether a human is looking at the TV/Monitor or not.
Re: It is not possible to detect and block Chrome headless
#108Chromium can work directly with Wayland afaik. Do a "fake" Wayland implementation and Chromium will happily think it's drawing crap
Re: It is not possible to detect and block Chrome headless
#109Earlier quoted context omitted.
In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…
> When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique. For one thing, detecting that a scraper is a scraper is the problem, not the prelude to the problem. You might as well block them at that point, if you feel you can reliably dete…
> I used to use massive amounts of crawled web data to accurately forecast earnings announcements months in advance.
This is really interesting. Was it using sentiment analysis from news-like sources, or tracking prices/releases, or some other information?
Re: It is not possible to detect and block Chrome headless
#110Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…
In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…
> Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would.
Honest question, would this open the site up to legal liability?
You can never identify a bot 100%, so if you intentionally provided false information to an otherwise legitimate user, and that user is harmed by your false information, isn't that a breach of contract that would make you liable for damages based on the harm your bad data caused?
What if the user claimed that the site was acting maliciously, providing lower prices to cause a market reaction ala what happened to coinmarketcap? How would you prove that you were only targeting bots or provide statistics that your methods were effective instead of arbitrary. That stuff matters when money is lost.
You'd have to have a clause in your terms of service that says "if we think you are a bot then the information we give you will be bad", then users could (and probably should) run far away, since they have no way of knowing whether you think they are a bot or not, and thus can't trust anything on your site?