11–20 of 65 posts
For the past couple of years, I've always done any web scraping with trusty python + beautiful soup or elementtree. I've recently started doing it with Clojure + Enlive (mostly as an excuse to use clojure for less academic excercises) but I really like it. From that perspective, Upton looks pretty cool, especially the debug mode.
Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.
There's phantomjs for that. Though I've actually had the most success with perl's WWW::Mechanize::Firefox and a headless X server.
If you insist on controlling webkit through javascript, you can use gnome-seedjs. But this is problematic/annoying because there's no commonjs implementation yet.. in phantomjs you can require() node modules in the outside context. Not so much in gnome-seedjs..
Also, X means it's not actually headless, even if you're using xserver-xorg-video-dummy or xvfb. For this reason, phantomjs got rid of the X requirement a number of versions ago.
Does Upton have any way of dealing with JavaScript or ajax calls? For a lot of the scraping I do in Python, this is crucial for me. I use Selenium's Webdriver (along with beautiful soup or lxml) now for that - definitely open to other options.
Nope. Sorry. :) While I'm sure this would be possible, I think a node.js scraper would probably be a better fit for that sort of a project.
Node, by itself, doesn't have full versions of a lot of the objects that Javascript on the pages would refer to (DOM, event model, etc.); phantomjs gives you all of that.
Link: http://import.io/
Is web scraping legal?
Is web scraping legal?
Earlier quoted context omitted.
Nope. Sorry. :) While I'm sure this would be possible, I think a node.js scraper would probably be a better fit for that sort of a project.
Perhaps better phantomjs ( http://phantomjs.org ), perhaps using casper ( http://casperjs.org ) on top to handle some of the glue code. Phantomjs is a full headless browser (it'll give you screenshots of the pages it downloads if you want them); casper is a library that makes sequencing tasks somewhat easier. Node, by itself, doesn't have full versions of a lot of the objects that Javascript on the pages would refer…
We maintain a Node to Phantom bridge for this: https://github.com/baudehlo/node-phantom-simple
Seems like it's not being actively developed, but again, never had a problem.
Edit: realize this is focused on single page scraping with data extraction. Could use them nicely together in fact.