Ask HN: Web Scraping with Node.js

#1

Post by ErikAugust » Mon, Jun 20, 2016, 6:36 PM UTC

Anyone out there have a cachet of good Node web scraping libs, and/or other resources? Opinions?

I have the intuition that my Google searches are a bit inadequate. Thank you!

Re: Ask HN: Web Scraping with Node.js

#2

Post by pesfandiar » Mon, Jun 20, 2016, 7:58 PM UTC

Should it be necessarily with Node.js? I've tried to find a good toolset in JS, but it seems other languages (e.g. Python and Java) are a lot more mature in scraping.

Re: Ask HN: Web Scraping with Node.js

#3

Post by jneumann004 » Mon, Jun 20, 2016, 8:36 PM UTC

I haven't used this for web scraping, but have you tried out http://phantomjs.org/. It's a great tool that lets your process webpages; I don't think it would be that difficult to set it up to scrape sites.

Re: Ask HN: Web Scraping with Node.js

#4

Post by Nilef » Mon, Jun 20, 2016, 11:13 PM UTC

Try this out https://github.com/IonicaBizau/scrape-it

Re: Ask HN: Web Scraping with Node.js

#5

Post by iends » Tue, Jun 21, 2016, 1:03 AM UTC

Check out https://github.com/cheeriojs/cheerio

Re: Ask HN: Web Scraping with Node.js

#6

Post by tmpfs » Tue, Jun 21, 2016, 1:12 AM UTC

I found https://github.com/cgiffard/node-simplecrawler to be good, I used it for https://github.com/tmpfs/linkdown - a cli for validating web pages etc.

Re: Ask HN: Web Scraping with Node.js

#7

Post by paulzerkel » Tue, Jun 21, 2016, 2:17 AM UTC

Here is a place to start: https://github.com/lorien/awesome-web-scraping/blob/master/j...

You might want to check out the Nightmare project (https://github.com/segmentio/nightmare) which makes automating an Electron instance pretty simple.