Cool, I think the new captchas use mouse entropy that would be an interesting test since remote usually go straight to the pixel point.
It is not possible to detect and block Chrome headless
71–80 of 174 posts
Re: It is not possible to detect and block Chrome headless
#72All those tests are useless and effective only against script kiddys (which are now like 99.99999% of developers by old standards) and are unable to code anything else but crappy languages like js. For people grown up with web, capable of coding in c/c++ those tests are a joke, I'll just modify the source code to return what is expected and 'game over'. We were reversing drms by dissasembling and patching the binaries - in world of text based protocols and scripts, Idiocracy of todays world is making us invincible.
Re: It is not possible to detect and block Chrome headless
#73Earlier quoted context omitted.
So record actual user input data and generate similar input patterns stochastically. That said if you try to scale this up beyond what a reasonable, normal user world do in one sitting, you are bound to stand out. Although that said, I find that I trigger such rate-limiting mechanisms already as a human just when searching Google as a human being and clicking through every last search result page.
You'd have to scrape slowly to mimic a real slow user. Maybe at that point you'd be cheaper to get Mechanical Turk to do it. That should solve IP rate limiting, captchas, and just about everything except the endless arms race. Why are so many people going directly to these same-formatted internal URLs without clicking through from random other places? So the site can change the internal URLs and break it all over aga…
>You'd have to scrape slowly to mimic a real slow user.
Sure, but that's easily mitigated by running multiple scrapers as different users.. You don't need to get all the data from a single scrape.
Re: It is not possible to detect and block Chrome headless
#74Crawlers & scrapers that rely on headless browsers like Chrome often initiate playback of video on the pages they access. The company I work for (Mux) has a product that collects user-experience metrics for video playback in browsers & native apps. It's been a non-trivial effort developing a system to identify video views from headless browsers so that we might limit their impact on metrics. Being able to make this d…
Video should never play unless actively initiated by the user. That would fix the metrics, as the headless browser probably wouldn't initiate the video playback
If it's a video site, I expect the video to play when I land, e.g. youtube. I'm initiating on purpose by browsing
Re: It is not possible to detect and block Chrome headless
#75Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…
In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…
> When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it.
Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique.
For one thing, detecting that a scraper is a scraper is the problem, not the prelude to the problem. You might as well block them at that point, if you feel you can reliably detect them. If nothing else it's more resource efficient than sending them fake data.
Second, and more importantly, "poisoning the well" is not going to work against a sophisticated scraper. I used to use massive amounts of crawled web data to accurately forecast earnings announcements months in advance. I've also consulted with various companies building distributed crawling systems or looking for ways to develop integrations without public APIs.
My colleagues would know very quickly if something was wrong with the data because our model would suddenly be extremely out of whack in ways that could be traced back to the website's behavior. We used to specifically look for this sort of thing and basically eyeball the data on a daily basis. We even had tools in place that measured the volume, response time and type of data being received, and would alert us if the ingested data went more than one standard deviation outside of the expectation in any of these metrics.
You might succeed in screwing up whatever the data is being used to inform for a little while, but you will, with near certainty, show your hand by doing this, and scrapers will react in the usual cat and mouse game. Modern scraping systems are extremely sophisticated and success stories are prone to confirmation bias, because you're mostly unaware when it's happening successfully on your website. For a while I was experimenting with borrowing (non-malicious) methods from remote timing attacks to identify when servers were treating automated requests differently from manual requests instead of simply dropping them. The rabbit hole of complexity is sufficiently deep that you could saturate a full research time with work to do in this area.
If you want to productively block scrapers, you should consider using a captcha-based system at the application layer, preferably a captcha that hasn't been broken yet and which can't be outsourced to a mechanical turk-based API. If nothing else, doing that will introduce at least 10 - 20 seconds of latency per request, which might be intolerable for many scrapers even if they're quite sophisticated.
Re: It is not possible to detect and block Chrome headless
#76Earlier quoted context omitted.
In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…
Poisoning the well is very effective. We employed it at a large ecommerce company that was getting hit by carders testing credit cards on low price point items(sub $5). We were playing cat and mouse with them for six months. Found certain attributes about the browser that the botnet was using and fed them randomized success/fail responses. After two weeks of feeding them bad data, they left and never came back. They…
I think this is a good example of "poisoning the well" in practice, and I was in a similar position as you describe when I was working in incident response at a consumer bank a few years ago.
That said, this is a very particular scenario, and I don't think you can generalize the effectiveness of the technique from this example. In situations where attackers are looking for boolean responses, i.e. to verify email addresses, credit cards, usernames, etc, this can work well.
But in most situations a crawler is looking for particular data. Moreover, it will have a strong expectation (and understanding) about what this data should look like. Poisoning the well is not going to work as well in that scenario, because a scraper will be aware that the data is incorrect.
In contrast, people testing credit cards don't need your data, they just need data from somewhere that verifies the card numbers. In that case it's easier to just keep rotating targets until they've tested all the cards instead of bootstrapping an in-house team to write bespoke crawlers.
Re: It is not possible to detect and block Chrome headless
#77All those tests are useless and effective only against script kiddys (which are now like 99.99999% of developers by old standards) and are unable to code anything else but crappy languages like js. For people grown up with web, capable of coding in c/c++ those tests are a joke, I'll just modify the source code to return what is expected and 'game over'. We were reversing drms by dissasembling and patching the binarie…
What does C/C++ have to do with this, when the point of the article is showing that they can be defeated using JS?
Re: It is not possible to detect and block Chrome headless
#78Isn't it impossible to win the game of blocking headless browsers? What's stopping someone from creating an API that opens up a real browser, uses a real (or virtual) keyboard, types in/clicks the real address, etc. then proceeds to use computer vision to scrape the information from the page without touching the DOM?
There's a similar "analog hole" for video DRM, too.
Re: It is not possible to detect and block Chrome headless
#79This article is a joke, all those methods of "protections" are a joke. What we called "script kiddys" and are now a major amount of so called developers are just underdeveloped lamers who just don't know that the fight is lost in advance. All the methods that you take are useless when you get into situation of scraper run by someone who is able to modify (oh and is able to code in c/c++) and recompile the client side. The world went into Idiocracy so much that methods are beeing invented by people who are so narrow minded that they see the development in a scope of a browser and have a false sense that they "can handle it". Only if the oponent is as narrow minded as they are. Only than. I can modify the source code of chromium, you will get back exactly what you expect from regular user, i am able to scrape fb and linkedin and the only thing they can do is to slow me down (to hide the fact that the code is doing surfing, not human). Stop wasting your time on protection, you are running your inneficient crappy code in insecure environment, the only "attacker" you are safe against is the one who is as clueless as you are.
The same moment when you send content to the client, it is game over. You have lost all control.
I am sorry for all non-gentle sentences here, but we had developers who were able to decompile asm code and patch it to avoid drms, while now sandboxed idiots are thinking, they are smart. The whole dev. environment became toxic =/ And people are just to stupid to understand how stupid they are =/
Re: It is not possible to detect and block Chrome headless
#80I'm not sure why one wants to bother to do this. With tools like Sikuli script (sikuli.org) already around for ages, automating a headed browser isn't rocket science. So the best-case scenario for detecting headless browsers is "The bad guys just use headed browsers and another automation solution."
Looks like great tool, never heard of sikuli before. Thanks for the tip!