What separates this from Nokogiri? (Don't take that as critical, more working code out in the world is better. Just wondering, as I use Nokogiri heavily for our company chat-bot, and couldn't tell the answer at a quick glance.)
Upton: A Web Scraping Framework
31–40 of 65 posts
Re: Upton: A Web Scraping Framework
#32Earlier quoted context omitted.
If anything, it might be against TOS but ASAIK, breaking TOS is not illegal.
Who was the guy who got jail time for breaking a TOS? Think it was iPhone related.
Are you thinking of weev? http://en.wikipedia.org/wiki/Weev#AT.26T_data_breach
Re: Upton: A Web Scraping Framework
#33Earlier quoted context omitted.
There's phantomjs for that. Though I've actually had the most success with perl's WWW::Mechanize::Firefox and a headless X server.
You don't even need phantomjs, really. You can use python + webkitgtk+ through the gobject bindings. The problem with phantomjs is that it's using an old version of webkit from an old version of qtwebkit from qt 4.8, whenever that was released. By comparison, webkitgtk+ can be compiled from upstream webkit whenever you please. If you insist on controlling webkit through javascript, you can use gnome-seedjs. But this…
Re: Upton: A Web Scraping Framework
#34What separates this from Nokogiri? (Don't take that as critical, more working code out in the world is better. Just wondering, as I use Nokogiri heavily for our company chat-bot, and couldn't tell the answer at a quick glance.)
Reposting a comment by the author from the article:
> Upton depends on Nokogiri, which is basically the BeautifulSoup port for Ruby.
> If you just used vanilla Nokogiri, you'd be responsible for writing code to fetch, save (maybe), debug and sew together all the pieces of your web scraper. Upton does a lot of that work for you, so you can skip the boilerplate.
Re: Upton: A Web Scraping Framework
#35Earlier quoted context omitted.
Perhaps better phantomjs ( http://phantomjs.org ), perhaps using casper ( http://casperjs.org ) on top to handle some of the glue code. Phantomjs is a full headless browser (it'll give you screenshots of the pages it downloads if you want them); casper is a library that makes sequencing tasks somewhat easier. Node, by itself, doesn't have full versions of a lot of the objects that Javascript on the pages would refer…
Or do what we do at Hubdoc, and use both Node and Phantom. Node for performance where it's possible, and Phantom where the site has been built in such a way that scraping in Node becomes not worth the effort of figuring out all the weird stuff they've done in client side JS. We maintain a Node to Phantom bridge for this: https://github.com/baudehlo/node-phantom-simple
Curious, you use the webserver module in phantomjs, is that right? And that's how you do the inter-process communication? I'm curious how you chose that over websockets, or over HTTP polling from your phantomjs client against a local node server..
What about using something like node-gir, or whatever appjs does to combine the event loops of node/v8 and chromium/v8?
Re: Upton: A Web Scraping Framework
#36Earlier quoted context omitted.
There's phantomjs for that. Though I've actually had the most success with perl's WWW::Mechanize::Firefox and a headless X server.
You don't even need phantomjs, really. You can use python + webkitgtk+ through the gobject bindings. The problem with phantomjs is that it's using an old version of webkit from an old version of qtwebkit from qt 4.8, whenever that was released. By comparison, webkitgtk+ can be compiled from upstream webkit whenever you please. If you insist on controlling webkit through javascript, you can use gnome-seedjs. But this…
Do you have any other resources about using webkitgtk with python for this sort of purpose?
Re: Upton: A Web Scraping Framework
#37Sounds pretty cool. Also, for those operating in "java land" there are Commons HttpClient[1] and Apache Tika[2] which, together, are a pretty potent combination for scraping web data.
Re: Upton: A Web Scraping Framework
#38Earlier quoted context omitted.
Nope. Sorry. :) While I'm sure this would be possible, I think a node.js scraper would probably be a better fit for that sort of a project.
Perhaps better phantomjs ( http://phantomjs.org ), perhaps using casper ( http://casperjs.org ) on top to handle some of the glue code. Phantomjs is a full headless browser (it'll give you screenshots of the pages it downloads if you want them); casper is a library that makes sequencing tasks somewhat easier. Node, by itself, doesn't have full versions of a lot of the objects that Javascript on the pages would refer…
I wrote a backend system using Phantom and Akka to generate graphs using D3 and rasterize them into PNGs and put them into user-specific emails.
Phantom has some quirks but overall it's pretty solid.
Re: Upton: A Web Scraping Framework
#39Looks like a very nice integrated solution for data scientists and researchers. It's interesting to see the different shapes tools like this take, depending on their target users. I've been happy with node.js + cheerio[1] and it's simplistic jquery-like API: request 'http://website.com/list_of_stories.html', (err, body) -> $ = cheerio.load(body) callback $('#comments li a.commenter-name').map(cheerio::text) Plus if y…
Re: Upton: A Web Scraping Framework
#40PHP SimpleDOM Class makes scraping just like working with jQuery on the backend. http://simplehtmldom.sourceforge.net/