Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It is not possible to detect and block Chrome headless

intoli.com

Post reply on HN
Print view View original

121–130 of 174 posts

Re: It is not possible to detect and block Chrome headless

#121

Post by odammit »

Blocking crawlers is dead simple:

Find a way to build an API for your data that allows you both to make money.

Any effort besides that is wasted.

Honey pots links? Great my crawler only clicks things that are visible. See capybara.

IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residential IPs (bought through squid, hello or anyone else) and a mix of tor nodes ( I sample your site with that too) to make sure I never approach that number. But then I also geolocate the IP so it’s only crawling during sensible browsing hours for that location.

Keystrokes detection? yeah I slow down keystrokes so it looks like Grandma is browsing

Mouse detection? looks like Michael j Fox is on your site (that’s an old Dell or Gateway Commercial reference don’t be mad)

Poison the well? I get a page from multiple IPs and headless browser combinations on different screen orientations and if I detect odd changes in data I flag that URL for a turk to provide insight/tune the crawler.

I keep the screenshot and full payload (css,js,html) that I use over time to do more devious shit like render old versions of your page behind a private nginx server so I can re-extract pieces of data I may have missed.

Stop trying to stop the crawling and figure out how to create a revenue stream.

Re: It is not possible to detect and block Chrome headless

#123

Post by aplorbust »

Its trivial to randomise HTTP headers, both the content and the order. There are free and commercial databases of user-agent strings available to any user, the same ones the websites may use.

Users can also modify or delete HTTP headers through local proxies, using the same proxy software that many high volume websites use. Sites that rely on redirects to set headers make this even easier.

p0f only works with TCP. Could this be another a selling point for alternative congestion controlled reliable transports that are not TCP, e.g. CurveCP? I have prototype "websites" on my local LAN that do not use TCP.

The arguments in favor of controlling access to public information through "secret hacker ninja shit" (https://news.ycombinator.com/item?id=16176572) are not winning on the www or in the courts. Consider the recent Oracle ruling and the pending LinkedIn HiQ case.

If the information is intended to be non-public, then there is no excuse for not using access controls. Anything from basic HTTP authentication to requiring client x509 certificates would suffice for making a believable claim.

Detecting headless Chrome and serving fake information, or any other such "secret hacker ninja shit" is not going to suffice as a legitimate access control, whether in practice or in an argument to a reasonable person.

The fact is in 2017 websites still cannot even tell what "browser" I am using, let alone what "device" I am using. They still get it wrong every time. Best they can do is make lousy guesses and block indiscriminately. Everything that is not what they want/expect is a "bot", a competitor, an evil villan. Yet they have no idea. Sometimes, assumptions need to be tested.1

  1 https://news.ycombinator.com/item?id=16103235 (where developer thought spike in traffic was an "attack")

Re: It is not possible to detect and block Chrome headless

#124

Post by odammit »

odammit wrote: post #121

Blocking crawlers is dead simple: Find a way to build an API for your data that allows you both to make money. Any effort besides that is wasted. Honey pots links? Great my crawler only clicks things that are visible. See capybara. IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residentia…

Caveat to my wasted effort comment:

Your'e an e-commerce site that has a problem with people buying goods (especially virtual goods, ebooks, gift cards, etc)[1] with stolen credit cards. You need a solution.

The hardest thing I've ever had to crawl (as I mention in another comment in this thread) has been linkedin and Facebook. Why? Because I have to be logged in to get the data I want.

If you want to stop crawlers you can also put the good stuff behind auth, but you need a solid auth mechanism. You can't just do email verification because I'll use shit like https://www.mailinator.com to generate a ton of fake emails to sign up for your site.

[1] Why virtual goods? You can't stop shipping or track down the person once the card is reported stolen.

Re: It is not possible to detect and block Chrome headless

#125

Post by vivekseth »

odammit wrote: post #121

Blocking crawlers is dead simple: Find a way to build an API for your data that allows you both to make money. Any effort besides that is wasted. Honey pots links? Great my crawler only clicks things that are visible. See capybara. IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residentia…

How would this stop crawling by users who cannot afford the API subscription or don't want to pay for it?

I think your suggestion would reduce crawling, but not prevent or block it.

Re: It is not possible to detect and block Chrome headless

#126

Post by thaumaturgy »

lr4444lr wrote: post #45
thaumaturgy wrote: post #21

Earlier quoted context omitted.

I now work for a company that is gathering metadata on the IP address space (in an effort to reduce the amount of abuse that sites and service providers have to deal with). It won't be very long before it'll be possible to identify most of the common proxying networks and block those. Scrapers can respond by setting up something like an ssh tunnel from a residential high speed connection to a remote server (so that s…

Just out of curiosity, how are you going to handle Luminati? They're bot-networking home users all over the world in exchange for free VPN.

That's pretty neat.

For our purposes, those residential IPs that are used maliciously through this service will just hurt the reputation of the ISP they belong to. I suspect we'll see something shake out in the data where this activity is limited to some kind of specific demographics (ISP, netrange, geographic location) and shouldn't interfere too badly with the system as a whole.

Re: It is not possible to detect and block Chrome headless

#127

Post by bpicolo »

chriswarbo wrote: post #103
bpicolo wrote: post #74

Earlier quoted context omitted.

If it's a video site, I expect the video to play when I land, e.g. youtube. I'm initiating on purpose by browsing

The first thing I do when hitting a YouTube URL is stop the video. Then I'll either run youtube-dl on the URL, or just paste it straight into a proper video player (VLC).

Pretty confident you're in the extreme minority on that one

Re: It is not possible to detect and block Chrome headless

#128

Post by odammit »

vivekseth wrote: post #125
odammit wrote: post #121

Blocking crawlers is dead simple: Find a way to build an API for your data that allows you both to make money. Any effort besides that is wasted. Honey pots links? Great my crawler only clicks things that are visible. See capybara. IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residentia…

How would this stop crawling by users who cannot afford the API subscription or don't want to pay for it? I think your suggestion would reduce crawling, but not prevent or block it.

Build a revshare API.

I assume the people that can't afford it are using it for non-revenue generating purposes. I don't know that I'd care about those people if they aren't taking away from my bottom line.

Re: It is not possible to detect and block Chrome headless

#129

Post by sushid »

vivekseth wrote: post #125
odammit wrote: post #121

Blocking crawlers is dead simple: Find a way to build an API for your data that allows you both to make money. Any effort besides that is wasted. Honey pots links? Great my crawler only clicks things that are visible. See capybara. IP thresholds? Great I have burner IPs that hit a good page of yours until I’m blocked (am I time banned, captchad or perma) and then I back that number out across my network of residentia…

How would this stop crawling by users who cannot afford the API subscription or don't want to pay for it? I think your suggestion would reduce crawling, but not prevent or block it.

Because you'd presumably need to authenticate (and have a paid account) to access the data.

Re: It is not possible to detect and block Chrome headless

#130

Post by odammit »

sushid wrote: post #129
vivekseth wrote: post #125

Earlier quoted context omitted.

How would this stop crawling by users who cannot afford the API subscription or don't want to pay for it? I think your suggestion would reduce crawling, but not prevent or block it.

Because you'd presumably need to authenticate (and have a paid account) to access the data.

I think they meant, since they can't afford the API, they'd just keep crawling your HTML. Combo move the API and putting your stuff behind auth and you solve a lot of the problem.

You've gotta get that good yummy content publicly accessible for google though, so you'll rank. So, that's a balancing act.

Post reply on HN