Use Node.js to Extract Data from the Web

This is cool... if the content is structured. (Ever tried finding addresses in arbitrary text? Much harder: http://smartystreets.com/products/liveaddress-api/extract)

Re: Use Node.js to Extract Data from the Web

#5

Post by aroman » Sun, Aug 25, 2013, 10:49 PM UTC

Cheerio is really really awesome. I've used it to build a considerably sophisticated web scraping backend to wrap my school's homework website and re-expose/augment via node/mongo/backbone/websockets.

There are definitely some bugs in cheerio if you're looking to do some really fancy selector queries, but for the most part it's extremely performant and pleasant to use.

If anyone is interested in seeing what a sophisticated, parallalized usage of cheerio looks like, feel free to browse through the app I was mentioning above -- it's open source: https://github.com/aroman/keeba/blob/master/jbha.coffee

Re: Use Node.js to Extract Data from the Web

#6

Post by httpteapot » Sun, Aug 25, 2013, 11:07 PM UTC

nodesocket wrote: post #2

Have you played around with node.io? https://github.com/chriso/node.io Encapsulates all this functionality in an easy to use interface.

Last commit 3 months ago. Do you know if this project is still alive?

Re: Use Node.js to Extract Data from the Web

#7

Post by level09 » Sun, Aug 25, 2013, 11:11 PM UTC

here is how I like to do it :

  from pyquery import PyQuery as pq
  doc = pq('http://google.com')
  print doc('#hplogo')

Re: Use Node.js to Extract Data from the Web

#8

Post by babby » Sun, Aug 25, 2013, 11:13 PM UTC

mholt wrote: post #4

This is cool... if the content is structured. (Ever tried finding addresses in arbitrary text? Much harder: http://smartystreets.com/products/liveaddress-api/extract )

Come on, that's not really a scraping problem, it's more of a text parsing problem coupled with an API lookup or scrape to verify the address.

Though, id probably just google for some good address regexes, match against pages, for each address just throw them into something like maps.google.com/?q=[address] then try to scrape whatever normally pops up for a valid result. Also helps if you're expecting addresses to be in a certain country.

Re: Use Node.js to Extract Data from the Web

#9

Post by hackula1 » Sun, Aug 25, 2013, 11:15 PM UTC

chatman wrote: post #3

Isn't scrapy easier to use than this?

Cheerio is really easy for anyone familiar with jQuery (most node.js devs I would imagine).

Re: Use Node.js to Extract Data from the Web

#10

Post by level09 » Sun, Aug 25, 2013, 11:16 PM UTC

chatman wrote: post #3

Isn't scrapy easier to use than this?

its probably more organized and easier to read than a huge number of nested callbacks