Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Detecting PhantomJS-based visitors

engineering.shapesecurity.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Detecting PhantomJS-based visitors

#21

Post by logn »

I'm currently writing a Java-based headless browser, https://github.com/MachinePublishers/jBrowserDriver

Beware it's Affero GPL.

It implements most of the Selenium WebDriver APIs. Currently it's a work in progress especially in regard to persistent tracking. But it's capable of addressing many issues raised in the article. As a bonus, for Java users it's a lot faster than GhostDriver at least.

Header ordering: https://github.com/MachinePublishers/jBrowserDriver/blob/mas...

DOM objects (user-agent, navigator, Canvas, Date) are addressed by injecting the response content: https://github.com/MachinePublishers/jBrowserDriver/blob/mas...

JS Engine: about 1 year newer than PhantomJS. Java ships with Qt5 WebKit.

I'll have some significant updates in the next couple of days surrounding all these issues.

Re: Detecting PhantomJS-based visitors

#22

Post by hwh »

andrenotgiant wrote: post #3

The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or…

I can trigger Google's Captcha simply by using Chrome's Incognito Mode plus a SSH-based SOCKS proxy. No automation at all. Which is probably against the "normal usage profile" build for data Google gets from me despite Incognito Mode. Well, for starters, dropping Cookies all the time is probably an indicator. Not sure if they can somehow see my SOCKS tunnel - I hope they can't, but am not basing my security on that.

Re: Detecting PhantomJS-based visitors

#23

Post by andrenotgiant »

curiously wrote: post #13
andrenotgiant wrote: post #3

The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or…

I think that is overestimating their efforts. To build such profile and to monitor each and every activity would be very costly. I think what happens is that when they detect too many requests from an ip address in a short interval amount of time they throw a captcha.

There has to be some calibration or customization beyond IP. Traffic from University IP vs my home looks very different.

Re: Detecting PhantomJS-based visitors

#24

Post by discreditable »

homakov wrote: post #11

These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"

> These techniques are extremely easy to bypass.

Especially considering nowadays you can automate IE with PowerShell [1] . I've used it myself to automate account creation on a legacy system at work.

[1] http://www.youdidwhatwithtsql.com/automating-internet-explor...

Re: Detecting PhantomJS-based visitors

#25

Post by codezero »

hwh wrote: post #22
andrenotgiant wrote: post #3

The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or…

I can trigger Google's Captcha simply by using Chrome's Incognito Mode plus a SSH-based SOCKS proxy. No automation at all. Which is probably against the "normal usage profile" build for data Google gets from me despite Incognito Mode. Well, for starters, dropping Cookies all the time is probably an indicator. Not sure if they can somehow see my SOCKS tunnel - I hope they can't, but am not basing my security on that.

I'm assuming that your SOCKS tunnel endpoint is on a VPS, or some hosted solution, if that's the case, this probably explains it since those IPs are not the source of typical "user" web traffic. I get the same when I use a VPN.

Re: Detecting PhantomJS-based visitors

#26

Post by Torgo »

discreditable wrote: post #24
homakov wrote: post #11

These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"

> These techniques are extremely easy to bypass. Especially considering nowadays you can automate IE with PowerShell [1] . I've used it myself to automate account creation on a legacy system at work. [1] http://www.youdidwhatwithtsql.com/automating-internet-explor...

It's been possible to do this via OLE interface and vbScript for over a decade. It's probably better now though, I seem to recall that it wasn't possible to automate clicking javascript alert buttons, and other annoying limitations.

Re: Detecting PhantomJS-based visitors

#27

Post by Torgo »

dsp1234 wrote: post #4
deloschang wrote: post #2

Interesting read on how to detect PhantomJS, particularly the stack trace detection. Though I'm curious whether these methods will still work if the adversary is using Selenium rather than a headless browser? In other words, what could be done to detect a Selenium instance over a human using the browser?

The first thing that comes to mind is that Selenium doesn't generate mouse move events when doing something like clicking two different buttons on a page, or when testing something like hover. So I would definitely concentrate on user-generated events, assuming that it's not easy to detect on the backend (via something like user-agent).

Pass a user agent saying you're a touchscreen browser.

Re: Detecting PhantomJS-based visitors

#29

Post by jacquesm »

logn wrote: post #21

I'm currently writing a Java-based headless browser, https://github.com/MachinePublishers/jBrowserDriver Beware it's Affero GPL. It implements most of the Selenium WebDriver APIs. Currently it's a work in progress especially in regard to persistent tracking. But it's capable of addressing many issues raised in the article. As a bonus, for Java users it's a lot faster than GhostDriver at least. Header ordering: https:…

That's a super piece of work! Thank you, I'll probably become one of your users.
Post reply on HN