Detecting PhantomJS-based visitors
engineering.shapesecurity.com
Detecting PhantomJS-based visitors
1–10 of 29 posts
Re: Detecting PhantomJS-based visitors
#2Interesting read on how to detect PhantomJS, particularly the stack trace detection. Though I'm curious whether these methods will still work if the adversary is using Selenium rather than a headless browser? In other words, what could be done to detect a Selenium instance over a human using the browser?
Re: Detecting PhantomJS-based visitors
#3The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google
In my experience, these two use detection algorithms that could best be described as "pattern-based."
Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or forced to fill out a CAPTCHA.
For example, you can hit Google Search once an hour via a painfully obvious scraper no problem. But, even if you take control of a Chrome browser via selenium, and write your scraper to do everything exactly like humans down to typing in the searches and moving the mouse around and clicking results, Google is ridiculously good at identifying bot vs human traffic patterns.
I think the algorithm builds a "normal usage profile" for a combination of IP,cookie/user,device and sets a threshold, activity above that threshold gets flagged.
Re: Detecting PhantomJS-based visitors
#4Interesting read on how to detect PhantomJS, particularly the stack trace detection. Though I'm curious whether these methods will still work if the adversary is using Selenium rather than a headless browser? In other words, what could be done to detect a Selenium instance over a human using the browser?
The first thing that comes to mind is that Selenium doesn't generate mouse move events when doing something like clicking two different buttons on a page, or when testing something like hover. So I would definitely concentrate on user-generated events, assuming that it's not easy to detect on the backend (via something like user-agent).
Re: Detecting PhantomJS-based visitors
#5With the norm nowadays using JS frameworks on the front-end I would expect more and more search engines to use things like PhantomJS to "scrape" the page. If you decide to reject traffic like this isn't there a good chance your SEO will fall?
Re: Detecting PhantomJS-based visitors
#6With the norm nowadays using JS frameworks on the front-end I would expect more and more search engines to use things like PhantomJS to "scrape" the page. If you decide to reject traffic like this isn't there a good chance your SEO will fall?
Things like phantomjs, not phantomjs. PhantomJS is often used to automate sites against the TOS.
Re: Detecting PhantomJS-based visitors
#7With the norm nowadays using JS frameworks on the front-end I would expect more and more search engines to use things like PhantomJS to "scrape" the page. If you decide to reject traffic like this isn't there a good chance your SEO will fall?
I doubt any serious search engine is literally using PhantomJS, so I wouldn't see this as a huge problem.
Google is in all likelihood using V8 directly and is explicit about who it is. In fact, most legitimate search engine spiders are.
Re: Detecting PhantomJS-based visitors
#8The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or…
My guess is that the pattern is simply "too many requests from a single IP" plus a manually maintained white list.
When working from within the network of a large company that uses a proxy I saw the CAPTCHA regularly for some time.
Of course the CAPTCHA gets answered mostly correctly in this case, which could trigger manual inspection and finally addition to the white list.
Re: Detecting PhantomJS-based visitors
#9Interesting read on how to detect PhantomJS, particularly the stack trace detection. Though I'm curious whether these methods will still work if the adversary is using Selenium rather than a headless browser? In other words, what could be done to detect a Selenium instance over a human using the browser?
The first thing that comes to mind is that Selenium doesn't generate mouse move events when doing something like clicking two different buttons on a page, or when testing something like hover. So I would definitely concentrate on user-generated events, assuming that it's not easy to detect on the backend (via something like user-agent).
[deleted]
Re: Detecting PhantomJS-based visitors
#10The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or…
My guess is that the pattern is simply "too many requests from a single IP" plus a manually maintained white list. When working from within the network of a large company that uses a proxy I saw the CAPTCHA regularly for some time. Of course the CAPTCHA gets answered mostly correctly in this case, which could trigger manual inspection and finally addition to the white list.
I also think it doesn't have to be that complex.
I once scraped Google Search a few thousand times, in a few seconds, a few times, just for fun, and in a school (1000+ people). It was quite funny when everyone suddenly saw a captcha.
Then a month ago or so I scraped Google+ for some statistics, maybe 1-3k times in 2-3 minutes and nothing happened.
The point is that my node.js scraper was far from perfect. Just a fake Chrome user agent, nothing else.
If a large organization with a static IP has a lot of users, Google is clever enough to figure out that it is large and has a high average number of requests by e.g. seeing that many people use their distinct Google Accounts.