Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use Node.js to Extract Data from the Web

storminthecastle.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: Use Node.js to Extract Data from the Web

#11

Post by nacs »

httpteapot wrote: post #6
nodesocket wrote: post #2

Have you played around with node.io? https://github.com/chriso/node.io Encapsulates all this functionality in an easy to use interface.

Last commit 3 months ago. Do you know if this project is still alive?

Haven't used node.io but 3 months isn't that old.

Also, if you check the issues page for the project ( https://github.com/chriso/node.io/issues ), the author seems to be responding to any open issues with the latest comment by author being a month ago.

Re: Use Node.js to Extract Data from the Web

#13

Post by STRML »

Don't forget streams, the more `node.js` way to parse HTML:

    var http = require('http');
    var tr = require('trumpet')();
    var request = require('request');
    request.get('http://www.echojs.com")
      .pipe(tr.createReadStream("article > span"))
      .pipe(process.stdout);
That's it! See https://github.com/substack/node-trumpet and their tests for more.

Re: Use Node.js to Extract Data from the Web

#16

Post by zenocon »

I've done a considerable amount of scraping; if you're poking around at nicely designed web pages, node/cheerio will be nice, but if you need to scrape data out of a DOM mess with quirks and iframes w/in iframes and forms buried 6 posts deep (inside iframes with quirks), I'd use PhantomJS + CasperJS. Having a real browser sometimes makes a difference.

Re: Use Node.js to Extract Data from the Web

#17

Post by chrisohara »

httpteapot wrote: post #6
nodesocket wrote: post #2

Have you played around with node.io? https://github.com/chriso/node.io Encapsulates all this functionality in an easy to use interface.

Last commit 3 months ago. Do you know if this project is still alive?

Author here.

Still active, although development has slowed down.

If you have any questions or issues just submit an issue @ Github and I'll help asap.

Re: Use Node.js to Extract Data from the Web

#18

Post by kanzure »

STRML wrote: post #13

Don't forget streams, the more `node.js` way to parse HTML: var http = require('http'); var tr = require('trumpet')(); var request = require('request'); request.get('http://www.echojs.com") .pipe(tr.createReadStream("article > span")) .pipe(process.stdout); That's it! See https://github.com/substack/node-trumpet and their tests for more.

And then there's hyperquest because maybe you want to do more than five simultaneous requests:

https://github.com/substack/hyperquest

Re: Use Node.js to Extract Data from the Web

#19

Post by ssafejava »

kanzure wrote: post #18
STRML wrote: post #13

Don't forget streams, the more `node.js` way to parse HTML: var http = require('http'); var tr = require('trumpet')(); var request = require('request'); request.get('http://www.echojs.com") .pipe(tr.createReadStream("article > span")) .pipe(process.stdout); That's it! See https://github.com/substack/node-trumpet and their tests for more.

And then there's hyperquest because maybe you want to do more than five simultaneous requests: https://github.com/substack/hyperquest

True - you can also disable the globalAgent or change the number of pooled connections. Connection pooling was generally a bad idea (tm) in Node and afaik will be removed in the near future.
Post reply on HN