That doesn't seem like a very safe thing to do... dont they have sc for PhantomJS one can checksum and run ./configure > make > sudo make install?
PhantomJS is pretty big. IIRC, building it takes quite some time. I think they bundle webkit and the necessary parts of Qt, and you'd have to be out of your mind to build that from source if you can avoid it. Using official distribution packages would be a better idea, but their freshness can vary, especially on RHEL.
Fair enough if you want to get straight to doing what you were planning to do.
PhantomJS is brilliant, but Selenium is a questionable choice for this task. For some reason, the creators of Selenium have decided that passing HTTP status codes back through the API is and always will be outside the scope of their project. So if you request a page and it returns 404 you have no way to find out (other than using crude heuristics). This makes Selenium completely unusable for anything I would have use…
Aren't you stuck with JavaScript then? Sure, PhanthomJS is awesome, but Python is even in the title, so it's not just a side note.
No, phantomjs includes a webserver module. That's what ghostdriver uses to implement the WebDriver API and you can use it to implement a custom API that you call from Python. So you have to use JavaScript to implement the API, but you can use Python to implement your tests or web data extraction or whatever your actual task is.
Difficult edge cases are never a good reason not to support the 99.9% case. Also, phantomjs has access to all the information you want and the WebDriver API already has a capabilities negotiation facility. [Edit] Don't forget that the original URL is the only one supplied by the client of the API. It may be incorrect for very different reasons than all the other resources included by the page itself. That's why it is…
These aren't edge cases. They're asked about constantly. Most people are using Selenium because they care about everything on the page. Otherwise, your stdlib HTTP client would be sufficient. That aside, if PhantomJS already has the info, you can always fetch it with executeScript. If you do feel that strongly about the status code part though, I'd urge you to comment on the public draft of the W3C spec: http://www.w…
I believe you can't use execute because any JavaScript you supply runs inside the page. You don't have access to the phantomjs specific callbacks you need to intercept http traffic.
Well, I think the matter is a bit more complicated than that. When dealing with a full browser, you fetch a lot of resources. The status code for the first page fetch may be easily obtained, but your API gets very wonky as soon as you want to get status codes for all linked resources. Even if you managed that, any Ajax requests would complicate things, especially if they have deferred loading. And then you have WebSo…
Difficult edge cases are never a good reason not to support the 99.9% case. Also, phantomjs has access to all the information you want and the WebDriver API already has a capabilities negotiation facility. [Edit] Don't forget that the original URL is the only one supplied by the client of the API. It may be incorrect for very different reasons than all the other resources included by the page itself. That's why it is…
To follow up to your edit, that may be true in one case. But it's perfectly reasonable to navigate via clicking, anything in the navigate API, JS actions, meta refreshes, and so on. Even in that one case, most people would expect redirects to be followed and basic auth protected pages to submit. Again, all tractable problems, but ones that are likely better handled by an interstitial layer where you can see the entire chain of requests & responses.
These aren't edge cases. They're asked about constantly. Most people are using Selenium because they care about everything on the page. Otherwise, your stdlib HTTP client would be sufficient. That aside, if PhantomJS already has the info, you can always fetch it with executeScript. If you do feel that strongly about the status code part though, I'd urge you to comment on the public draft of the W3C spec: http://www.w…
I believe you can't use execute because any JavaScript you supply runs inside the page. You don't have access to the phantomjs specific callbacks you need to intercept http traffic.
That's unfortunate. I don't work on PhantomJS, but I can try to track down someone on the team and see if there's a way to attach a handle to window or something.
From the point of view of simulating actual users, the fact that some random third-party resource on the page failed to load is not particularly relevant. That happens all the time as I browse around the web, and I never have to care about it as long as the site continues to function. So it very much is an edge case compared to the page itself failing to load.
A JavaScript file failing to load will bork most pages. A CSS file failing to load or a key image will cause most people to quit. And an Ajax request failing in a single-page app will render it useless. But, my point of view is from actual Selenium users. This is framed by providing support on the IRC channel, on the mailing lists, triaging the issue tracker, and by interacting with people at SeleniumConf and the loc…
"A JavaScript file failing to load will bork most pages. A CSS file failing to load or a key image will cause most people to quit."
Wha?
Sure, if, say, "app.js" fails to load, you have a problem.
But an analytics script?
A 3rd party ad script (which is what the GP gave as an example)?
Difficult edge cases are never a good reason not to support the 99.9% case. Also, phantomjs has access to all the information you want and the WebDriver API already has a capabilities negotiation facility. [Edit] Don't forget that the original URL is the only one supplied by the client of the API. It may be incorrect for very different reasons than all the other resources included by the page itself. That's why it is…
To follow up to your edit, that may be true in one case. But it's perfectly reasonable to navigate via clicking, anything in the navigate API, JS actions, meta refreshes, and so on. Even in that one case, most people would expect redirects to be followed and basic auth protected pages to submit. Again, all tractable problems, but ones that are likely better handled by an interstitial layer where you can see the entir…
Browsers do see the entire chain of requests and responses. All of it. Some browsers make that information available externally. I just don't see why a browser remote control solution like Selenium shouldn't pass on as much of this information as possible.
We do quite a bit of web scraping / parsing on headless servers with Selenium. What we did was just install some X packages and run VNC server on the headless clients with Firefox. Cool thing about that is you can then go watch the scripts executing if you connect to the VNC session and take a screenshot on failure, etc.
Brilliant! I've been using Xvfb for headless operation, didn't even consider using VNC.
I find it preferable to determine the requests that jQuery is making and perform them myself to extract the necessary data, rather than load up a whole browser just to do the same thing. Selenium is terrible , performance wise, and requires a significant investment in environment in order to work reliably. I try to avoid it except when I absolutely cannot.
I wound up doing this myself, after spending an undue amount of time struggling with a morass of insanely written Javascript. Fiddler proved indispensable for observing the actual interaction with the web server.