For anyone using PhantomJS I'd recommend checking out CasperJS (http://casperjs.org/) . It adds some nice features to PhantomJS and takes out a lot of the pain points
Python Headless Web Browser Scraping on Amazon Linux
11–20 of 40 posts
Re: Python Headless Web Browser Scraping on Amazon Linux
#12One more thing, has anyone used BeautifulSoup for forever? Is the project still active? I mean the website is cute and all, but I find pyquery ( Also based on lxml) so much easier with parsing the scraped data.
Re: Python Headless Web Browser Scraping on Amazon Linux
#13Earlier quoted context omitted.
Well, I think the matter is a bit more complicated than that. When dealing with a full browser, you fetch a lot of resources. The status code for the first page fetch may be easily obtained, but your API gets very wonky as soon as you want to get status codes for all linked resources. Even if you managed that, any Ajax requests would complicate things, especially if they have deferred loading. And then you have WebSo…
Difficult edge cases are never a good reason not to support the 99.9% case. Also, phantomjs has access to all the information you want and the WebDriver API already has a capabilities negotiation facility. [Edit] Don't forget that the original URL is the only one supplied by the client of the API. It may be incorrect for very different reasons than all the other resources included by the page itself. That's why it is…
These aren't edge cases. They're asked about constantly. Most people are using Selenium because they care about everything on the page. Otherwise, your stdlib HTTP client would be sufficient.
That aside, if PhantomJS already has the info, you can always fetch it with executeScript.
If you do feel that strongly about the status code part though, I'd urge you to comment on the public draft of the W3C spec: http://www.w3.org/TR/webdriver/
Re: Python Headless Web Browser Scraping on Amazon Linux
#14If you're writing Python and need to do something like this, you could try using Phantompy, a Python port of PhantomJS: https://github.com/niwibe/phantompy
It's still "in an early stage of development" but it's on my list of libraries to keep an eye on for when I have time to tackle the JS-heavy sites of the world.
Re: Python Headless Web Browser Scraping on Amazon Linux
#15You are installing some devel-packages, but i don't see anything compiling? Does the selenium installation build native extensions? Then the commands should probably the other way round. Or is phantomjs compiling something on the first run? Minor nitpick: I don't think it is a good idea to copy a binary directly to /usr/bin, without a package manager. You could just put it into /opt and symlink to /usr/(local/)bin.
The file that he is fetching ( phantomjs-1.9.1-linux-x86_64.tar.bz2 ) is the executables for his platform, with some examples on usage and a readme.
That doesn't seem like a very safe thing to do...
dont they have sc for PhantomJS one can checksum and run ./configure > make > sudo make install?
Re: Python Headless Web Browser Scraping on Amazon Linux
#16One more thing, has anyone used BeautifulSoup for forever? Is the project still active? I mean the website is cute and all, but I find pyquery ( Also based on lxml) so much easier with parsing the scraped data.
I'd consider it still active, since it was updated on 2013-06-07:
https://pypi.python.org/pypi/beautifulsoup4
I prefer using lxml myself, since I like using XPath queries, but bs4 sometimes parses broken HTML better than any of the provided lxml parsers do.
Re: Python Headless Web Browser Scraping on Amazon Linux
#17I find it preferable to determine the requests that jQuery is making and perform them myself to extract the necessary data, rather than load up a whole browser just to do the same thing.
Selenium is terrible, performance wise, and requires a significant investment in environment in order to work reliably. I try to avoid it except when I absolutely cannot.
Re: Python Headless Web Browser Scraping on Amazon Linux
#18PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…
Aren't you stuck with JavaScript then? Sure, PhanthomJS is awesome, but Python is even in the title, so it's not just a side note.
Re: Python Headless Web Browser Scraping on Amazon Linux
#19For scraping phantomjs or casperjs is the best way to go but you will have to use some JavaScript [1]. Both give you access to everything a WebKit browser user does with either a Node-style callback syntax (phantomjs) or a procedural/promises-style syntax (casperjs). Easy to setup, simple to use and fast enough for scraping but only WebKit (for now).
For testing on browsers other than WebKit (or vendor specific WebKit edge cases) use Selenium. Harder to setup, more complex, probably faster (still slow for testing) but not limited to WebKit.
[1] Sorry folks but some JavaScript is required to programmatically interacting with the web - also need some HTML and CSS.
Re: Python Headless Web Browser Scraping on Amazon Linux
#20Earlier quoted context omitted.
Difficult edge cases are never a good reason not to support the 99.9% case. Also, phantomjs has access to all the information you want and the WebDriver API already has a capabilities negotiation facility. [Edit] Don't forget that the original URL is the only one supplied by the client of the API. It may be incorrect for very different reasons than all the other resources included by the page itself. That's why it is…
These aren't edge cases. They're asked about constantly. Most people are using Selenium because they care about everything on the page. Otherwise, your stdlib HTTP client would be sufficient. That aside, if PhantomJS already has the info, you can always fetch it with executeScript. If you do feel that strongly about the status code part though, I'd urge you to comment on the public draft of the W3C spec: http://www.w…
From the point of view of simulating actual users, the fact that some random third-party resource on the page failed to load is not particularly relevant. That happens all the time as I browse around the web, and I never have to care about it as long as the site continues to function. So it very much is an edge case compared to the page itself failing to load.