Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

131–140 of 174 posts

Re: It is not possible to detect and block Chrome headless

#131

Post by littlestymaar »

odammit wrote: post #124
odammit wrote: post #121

Blocking crawlers is dead simple: Find a way to build an API for your data that allows you both to make money. Any effort besides that is wasted. Honey pots links? Great my crawler only clicks things that are visible. See capybara. IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residentia…

Caveat to my wasted effort comment: Your'e an e-commerce site that has a problem with people buying goods (especially virtual goods, ebooks, gift cards, etc)[1] with stolen credit cards. You need a solution. The hardest thing I've ever had to crawl (as I mention in another comment in this thread) has been linkedin and Facebook. Why? Because I have to be logged in to get the data I want. If you want to stop crawlers y…

> [1] Why virtual goods? You can't stop shipping or track down the person once the card is reported stolen.

In the meantime, virtual good are also zero-cost : when you sell a ebook and the transaction is cancelled by the bank, you didn't lose anything, it's not like the buyer was willing to pay anyway.

Re: It is not possible to detect and block Chrome headless

#132

Post by Whitestrake »

JeremyBanks wrote: post #118
nybble41 wrote: post #89

Earlier quoted context omitted.

Regarding YouTube in particular, I tend to open up videos in background tabs for later viewing and find it very annoying that they start playing automatically before I get around to that tab. I did go there to watch the video—eventually. Just not the second that the page finishes loading. YMMV. A persistent setting to enable or disable auto-play would be ideal.

Chrome doesn't autoplay videos in background tabs until you focus them.

Firefox likewise has toggles for disabling autoplay in background tabs (on by default) and disabling autoplay completely (off by default).

Re: It is not possible to detect and block Chrome headless

#133

Post by londons_explore »

pwaai wrote: post #41

Some people seem to have figured out how to detect without relying on fingerprinting the browser. ex. Crunchbase but headless chrome shouldn't be possible to distinguish from a regular chrome browser. The only vector to block scraper is some sort of navigational awareness that deviates from a distribution curve + awareness of IP. but this comes at a great cost to hurting your own real vistors by taxing them with capt…

Thats what invisible recaptcha is for.

Only the users who compulsively clear cookies ever get bothered by it, and even then all they have to do is click a few photos of cars.

Re: It is not possible to detect and block Chrome headless

#134

Post by exikyut »

dsacco wrote: post #75
troels wrote: post #27

Earlier quoted context omitted.

In my experience, the most effective counter measure to scraping is not to block, but rather to poison the well. When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Instead you begin feeding plausible, but wrong data (like, add a random number to price). This will usually cause much more damage to the scraper than blocking would. Depending on your…

> When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique. For one thing, detecting that a scraper is a scraper is the problem, not the prelude to the problem. You might as well block them at that point, if you feel you can reliably dete…

> If you want to productively block scrapers, you should consider using a captcha-based system at the application layer, preferably a captcha that hasn't been broken yet and which can't be outsourced to a mechanical turk-based API. If nothing else, doing that will introduce at least 10 - 20 seconds of latency per request, which might be intolerable for many scrapers even if they're quite sophisticated.

I'm thinking of getting into scraping as a (weird) hobby and still doing high-level research in the field. The point I've quoted stands out to me as one of the biggest hurdles to leap.

Three things.

Firstly, I'm aware of human-based captcha-defeating systems. You describe captchas that "cannot be outsourced to a mechanical turk-based API". I'm wasn't aware such systems existed, that sounds scary. Do you have any examples?

Secondly, I'm a little confused by your mention that #1 will simply raise the latency of the request to 10-20 seconds instead of completely blocking it.

Thirdly, in the case where a captcha (eg, ReCAPTCHA) can be defeated by a captcha-filling service... well, such services are extremely cheap, and there's no per-account fee or overhead. It sounds plausible that one could simply sign up for the captcha-filling service multiple times, and submit captchas to defeat in parallel. Could this work?

Re: It is not possible to detect and block Chrome headless

#135

Post by bsder »

dsacco wrote: post #105
Retric wrote: post #97

Earlier quoted context omitted.

Choosing a believable alteration is just as important as detecting a browser. For price scrapping even 3% changes to prices can do a lot of damage and is much harder to detect than completely bogus data.

That's a fair point, and it's conceivable this could work. But if the scrapers have had a legitimate source of truth in your data for a meaningful amount of time before you pull the trigger on this, it will get significantly harder. In practice I'm talking about data which is very multidimensional and has more nuanced metrics than something like a price point. As an example, let's say a certain company exposes sequen…

> As an example, let's say a certain company exposes sequentially crawlable online orders without authentication.

Wow. That's starting to get really shady.

Note to self: All things facing the web in ecommerce must be cryptographically randomized. :(

Re: It is not possible to detect and block Chrome headless

#136

Post by baybal2 »

From my experience in the scene:

Bot mill people are very aware of headless browsers being an effortless solution to mimic a browser, but not that efficient.

The amount of ram and so a bots spends to do a single click can truly hurt their bottom line.

Top tier collectives I heard of use own C/C++ frameworks with hardcoded requests and challenge solvers, and in-depth knowledge of anti-botting anti-fraud techniques used by the opposing force. If DoubleClick finds a brand new performance profiling test, and send it out in the JS code in one in 1000 requests, expect those guys to detect it and crack it within 24 hours.

They have no objective of getting through captchas, just having their number of valid clicks in double digits.

Re: It is not possible to detect and block Chrome headless

#137

Post by dsacco »

bsder wrote: post #135
dsacco wrote: post #105

Earlier quoted context omitted.

That's a fair point, and it's conceivable this could work. But if the scrapers have had a legitimate source of truth in your data for a meaningful amount of time before you pull the trigger on this, it will get significantly harder. In practice I'm talking about data which is very multidimensional and has more nuanced metrics than something like a price point. As an example, let's say a certain company exposes sequen…

> As an example, let's say a certain company exposes sequentially crawlable online orders without authentication. Wow. That's starting to get really shady. Note to self: All things facing the web in ecommerce must be cryptographically randomized. :(

> Wow. That's starting to get really shady.

Yes, there is a massive amount of crawling that happens in the service of financial forecasting. Satellite imagery, drones, web pages, API endpoints, receipt data (free financial aggregators sell this), location data (free geolocation services sell this), purchase history (free email clients sell this), etc. This rabbit hole goes very deep. Some of it is actively sold by free services for "market research", some of it is collected from sources that don't bother with authentication or access control and make all of their data public.

> Note to self: All things facing the web in ecommerce must be cryptographically randomized. :(

The bigger issue is that sensitive information should be behind authentication. If you require authentication instead of just a "browsewrap" terms and conditions clause at the bottom of the page, that data becomes legally actionable if it's ever found. Otherwise you're relying on (at best) a robots.txt to do the enforcement for you.

Many companies overlook this, even if they're told about it, because it doesn't compromise users or constitute a security vulnerability. So savvy firms take the data as a competitive advantage and use it for market research.

Re: It is not possible to detect and block Chrome headless

#138

Post by lr4444lr »

thaumaturgy wrote: post #126
lr4444lr wrote: post #45

Earlier quoted context omitted.

Just out of curiosity, how are you going to handle Luminati? They're bot-networking home users all over the world in exchange for free VPN.

That's pretty neat. For our purposes, those residential IPs that are used maliciously through this service will just hurt the reputation of the ISP they belong to. I suspect we'll see something shake out in the data where this activity is limited to some kind of specific demographics (ISP, netrange, geographic location) and shouldn't interfere too badly with the system as a whole.

That's what I thought. In case you were wondering, as a Luminati user (on the proxy end, not the VPN end thank god), my experience has been higher latency and request fail rate than on traditional proxy networks. It's definitely not my first weapon in the arsenal, but since the target for multiple reasons already has an effective high latency and my users are sufficiently motivated enough to wait for what I'm delivering, it's a pretty effective last resort.

Re: It is not possible to detect and block Chrome headless

#139

Post by cbr »

jimrandomh wrote: post #6

Sites detecting headless browsers vs headless browsers trying not to be detected by sites, is an arms race that's been going on for a long time. The problem is that, if you're trying to detect headless browsers in order to stop scraping, you're stepping into an arms race that's being played very, very far above your level. The main context in which Javascript tries to detect whether it's being run headless is when ma…

This is also the game people in click fraud vs click fraud detection are playing.

Re: It is not possible to detect and block Chrome headless

#140

Post by dsacco »

exikyut wrote: post #134
dsacco wrote: post #75

Earlier quoted context omitted.

> When you detect a scraper - through what ever means - you don't block it, as that would tip it off that you are on to it. Having written bespoke scraping systems professionally, I think you overestimate the applicability of this technique. For one thing, detecting that a scraper is a scraper is the problem, not the prelude to the problem. You might as well block them at that point, if you feel you can reliably dete…

> If you want to productively block scrapers, you should consider using a captcha-based system at the application layer, preferably a captcha that hasn't been broken yet and which can't be outsourced to a mechanical turk-based API. If nothing else, doing that will introduce at least 10 - 20 seconds of latency per request, which might be intolerable for many scrapers even if they're quite sophisticated. I'm thinking o…

> Firstly, I'm aware of human-based captcha-defeating systems. You describe captchas that "cannot be outsourced to a mechanical turk-based API". I'm wasn't aware such systems existed, that sounds scary. Do you have any examples?

Google's latest captcha specification (and similarly sophisticated systems) must be completed in a small window of time, change rapidly, have a varying number of "rounds", and are extremely antagonistic to being reloaded in e.g. a frame.

Practically speaking, you can't consistently outsource that to a third party API that uses humans to click and verify the images.

> Secondly, I'm a little confused by your mention that #1 will simply raise the latency of the request to 10-20 seconds instead of completely blocking it.

For captcha systems which can be defeated by humans, they will tolerate an extended period of time before they're solved. This means the amount of time it takes your crawler to send the data to the API and receive the challenge response, then send it to the captcha on the page will not cause the test to fail. It will, however, introduce a significant amount of latency in your requests.

> Thirdly, in the case where a captcha (eg, ReCAPTCHA) can be defeated by a captcha-filling service... well, such services are extremely cheap, and there's no per-account fee or overhead. It sounds plausible that one could simply sign up for the captcha-filling service multiple times, and submit captchas to defeat in parallel. Could this work?

Sure, but what are you gaining? These things are usually priced on a per-captcha basis. So you could have multiple accounts, but you'll be paying the same as just sending multiple requests to the API, and the concurrent requests are routed via the API's backend to different human operators for a solution.

Post reply on HN