These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"
Detecting PhantomJS-based visitors
11–20 of 29 posts
Re: Detecting PhantomJS-based visitors
#12With the norm nowadays using JS frameworks on the front-end I would expect more and more search engines to use things like PhantomJS to "scrape" the page. If you decide to reject traffic like this isn't there a good chance your SEO will fall?
I doubt any serious search engine is literally using PhantomJS, so I wouldn't see this as a huge problem. Google is in all likelihood using V8 directly and is explicit about who it is. In fact, most legitimate search engine spiders are.
I agree, I was speaking more on trying to block crawlers that look like browsers in general, sorry that wasn't more clear.
Re: Detecting PhantomJS-based visitors
#13The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or…
I think that is overestimating their efforts. To build such profile and to monitor each and every activity would be very costly.
I think what happens is that when they detect too many requests from an ip address in a short interval amount of time they throw a captcha.
Re: Detecting PhantomJS-based visitors
#14These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"
I agree with homakov. The article itself highlights defenses which are ridiculously naive and easy to bypass. Worst case, you don't use phantomjs, most of these tricks to block will be useless.
Re: Detecting PhantomJS-based visitors
#15These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"
A week is a subset of eventually, so technically the statement is true.
On a serious note - the article mentions repeatedly that many of the techniques can be trivially defeated. The main purpose of the article was to highlight some interesting, unique properties of PhantomJS that most people are not aware of (e.g. stack trace analysis).
Re: Detecting PhantomJS-based visitors
#16These techniques are extremely easy to bypass. It's like confirm('are you bot?') for CAPTCHA protection. It will detect headless browsers until someone decides to bypass it with proper headless webkit or something else. "and a sophisticated adversary will get through eventually" -> "any adversary will get through it in a week max"
A week is a subset of eventually, so technically the statement is true. On a serious note - the article mentions repeatedly that many of the techniques can be trivially defeated. The main purpose of the article was to highlight some interesting, unique properties of PhantomJS that most people are not aware of (e.g. stack trace analysis).
I agree that highlighted properties are interesting, but if we're talking about practical usage of these techniques it makes almost no sense (benefits to efforts ratio is too low)
Re: Detecting PhantomJS-based visitors
#17Earlier quoted context omitted.
A week is a subset of eventually, so technically the statement is true. On a serious note - the article mentions repeatedly that many of the techniques can be trivially defeated. The main purpose of the article was to highlight some interesting, unique properties of PhantomJS that most people are not aware of (e.g. stack trace analysis).
I agree that highlighted properties are interesting, but if we're talking about practical usage of these techniques it makes almost no sense (benefits to efforts ratio is too low)
Efforts to detect a script kiddies (or anyone who didn't bother heavily modify PhantomJS/QtWebKit/WebKit) by looking at properties mentioned in the article are really nothing compared to other approaches people use these days.
Re: Detecting PhantomJS-based visitors
#18I think people may be surprised at the number of bots hitting their sites. Even if they're not directly malicious, they serve no useful purpose, they won't see your ads or sign up for your deal, and just inflate your traffic numbers (unless you find that desirable).
There are a number of ways to try to detect and trick them out. I've done some unusual things to attempt to detect them, finding people coming thru proxies as well.
Sure, these simple methods can be defeated by an adversary, but at least judging from my experience, even simple methods you're going to catch some of them. Some hackers ain't that smart. A little surpised Cloudflare for instance wouldn't offer this (PhantomJS blocking), if they don't already. I'm all for hacking, but a site owner you should have the right to block bots if you so choose.
Re: Detecting PhantomJS-based visitors
#19I think people may be surprised at the number of bots hitting their sites. Even if they're not directly malicious, they serve no useful purpose, they won't see your ads or sign up for your deal, and just inflate your traffic numbers (unless you find that desirable). There are a number of ways to try to detect and trick them out. I've done some unusual things to attempt to detect them, finding people coming thru proxi…
If you don't want it crawled, don't put it on the public internet. You're just participating in an arms race.
Re: Detecting PhantomJS-based visitors
#20The companies that I know of that have a serious interest in bot detection and blocking are: Amazon and Google In my experience, these two use detection algorithms that could best be described as "pattern-based." Hitting any of their pages once via any scraper will not get you blocked, but exhibiting request patterns that sufficiently differentiate you from human traffic definitely will get you temporarily blocked or…
This type of protection is also aligned with what I think of as the most legitimate concern, which is resource exhaustion. I don't understand why some site operators object to scrapers even when they are well-behaved, and collecting public information. I think if the load is low, let scrapers scrape.