If you speak Ruby, mechanize is good: https://github.com/sparklemotion/mechanize
Ask HN: What are best tools for web scraping?
11–20 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#12If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…
I've recently made a little project with scrapy (for crawling) and BeautifulSoup (for parsing html) and it works out great. One more thing to add to the above list are pipelines, they make downloading files quite easy.
Re: Ask HN: What are best tools for web scraping?
#13The "best tool" is different for web developers and non-coders. If you are a non-technical person that just needs some data there is:
(1) hosted services like mozenda
(2) visual automation tools like Kantu Web Automation (which includes OCR)
(3) and last but not least outsourcing the scraping on sites like Freelancer.com
Re: Ask HN: What are best tools for web scraping?
#14Outwit Hub, specifically the advanced or enterprise levels. It has a GUI on it that is not designed very well, and documentation that is complete, but hard to search... But it can do just about any type of scrape, including getting started from a command line script
Second this. My go-to for years now. Inexpensive for what it does. Factor in the cost of building out it's features in your home rolled solution, and you'll be saving a ton. Plus the team is very responsive if you need support. And is open to small consulting projects if you need something beyond your own abilities.
Re: Ask HN: What are best tools for web scraping?
#15I would recommend using Headless Chrome along with a library like puppeteer[0]. You get the advantage of using a real browser with which you run pages' javascript, load custom extensions, etc.
Re: Ask HN: What are best tools for web scraping?
#16I've actually wrote about this! General tips that I've found from doing more than a few projects [0], and then an overview of Python libraries I use [1].
If you don't want to clock on the links, requests and BeautifulSoup / lxml is all you need 90% of the time. Throw gevent in there and you can get a lot of scraping done in not as much time as you think it would take.
And as long as we're talking about web scraping, I'm a huge fan of it. There's so much data out there that's not easily accessible and needs to be cleaned and organized. When running a learning algorithm, for example, a very hard part that isn't talked about a lot is getting the data before throwing it in a learning function or library. Of course, there the legal side of it if companies are not happy with people being able to scrape, but that's a different topic.
I'll keep going. The best way to learn about what are the best tools is to do a project on your own and teat them all out. Then you'll know what suits you. That's absolutely the best way to learn something about programming -- doing it instead of reading about it.
[0] https://bigishdata.com/2017/05/11/general-tips-for-web-scrap...
[1] https://bigishdata.com/2017/06/06/web-scraping-with-python-p...
Re: Ask HN: What are best tools for web scraping?
#17For very simple tasks Listly seems to be a fast and good solution: http://www.listly.io/
If you need more power, I heard good stuff about http://80legs.com/ though never tried them myself.
If you really need to do crazy shit like crawling the iOS App Store really fast and keep thing up to date. I suggest using Amazon Lambda and a custom Python parser. Though Lambda is not meant for this kind of things it works really well and is super scalable at a reasonable price.
Re: Ask HN: What are best tools for web scraping?
#18Depends on your skillset and the data you want to scrape. I am testing waters for a new business that relies on scraped data. As a non programmer I had good success testing stuff with contentgrabber. Import.io also get mentioned a lot. Tried out octoparse but wast stable with the scraping.
I find the desktop tool by import.io a little challenging to work with. Their toy web-demo is solid for simple table extraction, though.
Re: Ask HN: What are best tools for web scraping?
#19If you speak Ruby, mechanize is good: https://github.com/sparklemotion/mechanize
There is a Python version too: https://pypi.python.org/pypi/mechanize/ (And both are based on the perl version: http://search.cpan.org/dist/WWW-Mechanize/)
Re: Ask HN: What are best tools for web scraping?
#20Scrapy [https://github.com/scrapy/scrapy] works really well.