Python Headless Web Browser Scraping on Amazon Linux

#1

Post by steven5158 » Mon, Jun 17, 2013, 12:36 PM UTC

Python Headless Web Browser Scraping on Amazon Linux
fruchterco.com

Re: Python Headless Web Browser Scraping on Amazon Linux

#2

Post by phaer » Mon, Jun 17, 2013, 1:38 PM UTC

You are installing some devel-packages, but i don't see anything compiling? Does the selenium installation build native extensions? Then the commands should probably the other way round. Or is phantomjs compiling something on the first run?

Minor nitpick: I don't think it is a good idea to copy a binary directly to /usr/bin, without a package manager. You could just put it into /opt and symlink to /usr/(local/)bin.

Re: Python Headless Web Browser Scraping on Amazon Linux

#3

Post by 616c » Mon, Jun 17, 2013, 1:58 PM UTC

I recently tried to get back into Selenium for a work-related project and, despite its frustrations, it is one my favorite open source gems I found in the last several years. When showing it uninitiated web devs their heads almost exploded from joy and amazement. Your setup with Selenium intrigued me since the pain point for me has become how difficult it is to maneuver some browsers with Selenium IDE to throw together ideas, if that is even encouraged anymore.

Re: Python Headless Web Browser Scraping on Amazon Linux

#4

Post by fauigerzigerk » Mon, Jun 17, 2013, 1:59 PM UTC

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have used it for.

Fortunately you can do it by using phantomjs directly instead of going through the Selenium WebDriver API. Maybe one day the phantomjs WebDriver API implementation (ghostdriver) will extend the API to pass HTTP status information back to the caller. Until then, this API is unusable (at least for me).

Re: Python Headless Web Browser Scraping on Amazon Linux

#5

Post by kawsper » Mon, Jun 17, 2013, 2:23 PM UTC

phaer wrote: post #2

You are installing some devel-packages, but i don't see anything compiling? Does the selenium installation build native extensions? Then the commands should probably the other way round. Or is phantomjs compiling something on the first run? Minor nitpick: I don't think it is a good idea to copy a binary directly to /usr/bin, without a package manager. You could just put it into /opt and symlink to /usr/(local/)bin.

The file that he is fetching ( phantomjs-1.9.1-linux-x86_64.tar.bz2 ) is the executables for his platform, with some examples on usage and a readme.

Re: Python Headless Web Browser Scraping on Amazon Linux

#6

Post by ejk314 » Mon, Jun 17, 2013, 2:40 PM UTC

fauigerzigerk wrote: post #4

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…

You could always write a simple proxy in python and simply route all of your traffic through that.

See: http://voorloopnul.com/blog/a-python-proxy-in-less-than-100-...

Re: Python Headless Web Browser Scraping on Amazon Linux

#7

Post by fauigerzigerk » Mon, Jun 17, 2013, 2:49 PM UTC

ejk314 wrote: post #6

fauigerzigerk wrote: post #4

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…

You could always write a simple proxy in python and simply route all of your traffic through that. See: http://voorloopnul.com/blog/a-python-proxy-in-less-than-100-...

That would add quite a lot of complexity to achieve something rather trivial.

Re: Python Headless Web Browser Scraping on Amazon Linux

#8

Post by nirvdrum » Mon, Jun 17, 2013, 2:59 PM UTC

ejk314 wrote: post #6

fauigerzigerk wrote: post #4

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…

You could always write a simple proxy in python and simply route all of your traffic through that. See: http://voorloopnul.com/blog/a-python-proxy-in-less-than-100-...

BrowserMob Proxy is the go-to tool for use with Selenium:

http://bmp.lightbody.net/

Re: Python Headless Web Browser Scraping on Amazon Linux

#9

Post by nirvdrum » Mon, Jun 17, 2013, 3:02 PM UTC

fauigerzigerk wrote: post #4

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…

Well, I think the matter is a bit more complicated than that. When dealing with a full browser, you fetch a lot of resources. The status code for the first page fetch may be easily obtained, but your API gets very wonky as soon as you want to get status codes for all linked resources. Even if you managed that, any Ajax requests would complicate things, especially if they have deferred loading. And then you have WebSockets.

There are tools, such as BrowserMob Proxy, far better suited for monitoring HTTP traffic. And they'll get you all the headers. You can even capture to HAR so you measure performance.

Re: Python Headless Web Browser Scraping on Amazon Linux

#10

Post by fauigerzigerk » Mon, Jun 17, 2013, 3:07 PM UTC

nirvdrum wrote: post #9

fauigerzigerk wrote: post #4

PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…

Well, I think the matter is a bit more complicated than that. When dealing with a full browser, you fetch a lot of resources. The status code for the first page fetch may be easily obtained, but your API gets very wonky as soon as you want to get status codes for all linked resources. Even if you managed that, any Ajax requests would complicate things, especially if they have deferred loading. And then you have WebSo…

Difficult edge cases are never a good reason not to support the 99.9% case.

Also, phantomjs has access to all the information you want and the WebDriver API already has a capabilities negotiation facility.

[Edit] Don't forget that the original URL is the only one supplied by the client of the API. It may be incorrect for very different reasons than all the other resources included by the page itself. That's why it is justified to treat it as a special case.