21–30 of 35 posts
I've done a considerable amount of scraping; if you're poking around at nicely designed web pages, node/cheerio will be nice, but if you need to scrape data out of a DOM mess with quirks and iframes w/in iframes and forms buried 6 posts deep (inside iframes with quirks), I'd use PhantomJS + CasperJS. Having a real browser sometimes makes a difference.
Don't forget streams, the more `node.js` way to parse HTML: var http = require('http'); var tr = require('trumpet')(); var request = require('request'); request.get('http://www.echojs.com") .pipe(tr.createReadStream("article > span")) .pipe(process.stdout); That's it! See https://github.com/substack/node-trumpet and their tests for more.
var tr = require('trumpet')();
tr.createReadStream('article > span')
.pipe(process.stdout);
var request = require('request');
request.get('http://www.echojs.com').pipe(tr);
https://github.com/karlwestin/node-gumbo-parser
It might be interesting if someone were to implement a Cheerio-like API on top of that, as Cheerio has a nicer API but Gumbo's parser is more spec-compliant.
nice! I did a webcrawler with node.js myself last year. It's only a quick try but you can find the worker class here: https://gist.github.com/zerni/6337067 Unfortunately jsdom had a memory leak so the crawler died after a while...
I've done a considerable amount of scraping; if you're poking around at nicely designed web pages, node/cheerio will be nice, but if you need to scrape data out of a DOM mess with quirks and iframes w/in iframes and forms buried 6 posts deep (inside iframes with quirks), I'd use PhantomJS + CasperJS. Having a real browser sometimes makes a difference.
I've done a considerable amount of scraping; if you're poking around at nicely designed web pages, node/cheerio will be nice, but if you need to scrape data out of a DOM mess with quirks and iframes w/in iframes and forms buried 6 posts deep (inside iframes with quirks), I'd use PhantomJS + CasperJS. Having a real browser sometimes makes a difference.
I'm also looking at doing a web-scraping project with Node.js.
I was going to go with CasperJS (http://casperjs.org/), which seems fairly active and is based on PhantomJS.
Their quickstart guide is actually creating a scraper:
http://docs.casperjs.org/en/latest/quickstart.html
However, I'm wondering how this (Cheerio) compares - anybody have any experiences?
It was initially built as a hack project to replace a core subset of YQL. (I helped to guide an intern at my company Dharmafly, Aaron Acerboni, when he built it).