Headless Chrome, Puppeteer, NodeJS (jsdom), and MongoDB. Fantastic stack for web data mining. Async based using promises for explicit user input flow automation.
Ask HN: What are best tools for web scraping?
21–30 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#22A good host xD
Preferably one that doesn't mind giving you a bunch of IPs, and if they do, don't charge a fortune for them.
Then you can worry about what software you're gonna use.
Re: Ask HN: What are best tools for web scraping?
#23If you speak Ruby, mechanize is good: https://github.com/sparklemotion/mechanize
I remember trying to use mechanize as a beginning rubyist and I can't recommend it from that experience. Specifically I remember poor documentation and confusing layers of abstraction. It might be better now that I know what the DOM is and how jQuery selectors work, but my first impression was abysmal.
Re: Ask HN: What are best tools for web scraping?
#24I used CasperJS[0] in the past to scrap a javascript heavy forum (ProBoards) and it worked well. But that was a few years ago, I have no idea what new strategies came up in the meantime.
Re: Ask HN: What are best tools for web scraping?
#25I use nightmarejs https://github.com/segmentio/nightmare which is based on electron; I recommend it if you're on js
Re: Ask HN: What are best tools for web scraping?
#26I use R since that is the language I use mostly httr and rvest. Edit I missed typing rvest thanks for the comments you use the two together.
https://cran.r-project.org/web/packages/httr/vignettes/quick...
Re: Ask HN: What are best tools for web scraping?
#27Headless Chrome, Puppeteer, NodeJS (jsdom), and MongoDB. Fantastic stack for web data mining. Async based using promises for explicit user input flow automation.
I agree with headless chrome.
I have used it with a locally hosted extension to allow easy access to dom and JavaScript after load. Then dumped results to a node app. Was very happy with the results.
Re: Ask HN: What are best tools for web scraping?
#28For non-coders, import.io is great. However, they used to have a generous free plan that has since went away (you are limited to 500 records now). Still a great product, problem is they don't have a small plan (starts at $299/month and goes up to $9,999).
Re: Ask HN: What are best tools for web scraping?
#29If you are scraping specific pages on a site, curl. Then transform that into the language you use.
Re: Ask HN: What are best tools for web scraping?
#30This tool takes a list of URIs and crawls each site for contact info. Phone, email, twitter, etc