We also used cheerio and node.js and built an click & extract interface around it: http://www.site2mobile.com/.
Use Node.js to Extract Data from the Web
31–35 of 35 posts
Re: Use Node.js to Extract Data from the Web
#32nice! I did a webcrawler with node.js myself last year. It's only a quick try but you can find the worker class here: https://gist.github.com/zerni/6337067 Unfortunately jsdom had a memory leak so the crawler died after a while...
If you want to fix the memory leak, I remember you need to do `window.close()` after the job is done.
thanks mate!
Re: Use Node.js to Extract Data from the Web
#33I've done a considerable amount of scraping; if you're poking around at nicely designed web pages, node/cheerio will be nice, but if you need to scrape data out of a DOM mess with quirks and iframes w/in iframes and forms buried 6 posts deep (inside iframes with quirks), I'd use PhantomJS + CasperJS. Having a real browser sometimes makes a difference.
Does this help in scraping website which provide data via jquery ? I mean does this render the javascript on page ?
Yes. It interprets and executes JS like a real browser would. Which is nice. For Python: http://jeanphix.me/Ghost.py/
Re: Use Node.js to Extract Data from the Web
#34Similar to what I wrote a week ago: http://greenido.wordpress.com/2013/08/21/yahoo-finance-api-w...
:)
Re: Use Node.js to Extract Data from the Web
#35We also used cheerio and node.js and built an click & extract interface around it: http://www.site2mobile.com/ .
Interesting, I encounter the same set of problems as well last year when working on two side projects. Ended up building a webscraping service with a point and click interface on top of it : https://krake.io