Use cdp4j (https://github.com/webfolderio/cdp4j) for Java.
Ask HN: What are best tools for web scraping?
101–110 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#102Always fascinated by how diverse the discussion and answers is for HN threads on web-scraping. Goes to show that "web-scraping" has a ton of connotations, everything from automated-fetching of URLs via wget or cURL, to data management via something like scrapy.
Scrapy is a whole framework that may be worthwhile, but if I were just starting out for a specific task, I would use:
- requests http://docs.python-requests.org/en/master/
- lxml http://lxml.de/
- cssselect https://cssselect.readthedocs.io/en/latest/
Python 3, AFAIK, doesn't have anything as handy as Ruby/Perl's Mechanize. But using the web developer tools you can usually figure out the requests made by the browser and then use the Session object in the Requests library to deal with stateful requests:
http://docs.python-requests.org/en/master/user/advanced/
I usually just download pages/data/files as raw files and worry about parsing/collating them later. I try to focus on the HTTP mechanics and, if needed, the HTML parsing, before worrying about data extraction.
Re: Ask HN: What are best tools for web scraping?
#103Always fascinated by how diverse the discussion and answers is for HN threads on web-scraping. Goes to show that "web-scraping" has a ton of connotations, everything from automated-fetching of URLs via wget or cURL, to data management via something like scrapy. Scrapy is a whole framework that may be worthwhile, but if I were just starting out for a specific task, I would use: - requests http://docs.python-requests.o…
All great advice. I've written dozens of small purpose-built scrapers and I love your last point.
It's pretty much always a great idea to completely separate the parts that perform the HTTP fetches and the part that figures out what those payloads mean.
Re: Ask HN: What are best tools for web scraping?
#104Apify (https://www.apify.com) is a web scraping and automation platform where you can extract data from any website using a few simple lines of JavaScript. It's using headless browsers, so that people can extract data from pages that have complex structure, dynamic content or employ pagination.
Recently the platform added support for headless Chrome and Puppeteer, you can even run jobs written in Scrapy or any other library as long as it can be packaged as Docker container.
Disclaimer: I'm a co-founder of Apify
Re: Ask HN: What are best tools for web scraping?
#105I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service. All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/ / / ) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception. We have cronjob (cron + jenkins) that trigger dataset upda…
> We use Redis to send task (update / discovery) to our crawlers.
Some kind of queue implemented with Redis? How does it work?
Re: Ask HN: What are best tools for web scraping?
#106I've been doing scraping for many years and at the end it's always the same, you build a lot of stuff to bypass site restrictions and finally, once you are done, you can start scraping. It all goes fine until the site you are scrapping bans you...
So what I do now?
- proxycrawl https://proxycrawl.com
- node http://nodejs.org
With proxycrawl I don't need to worry about bans or blocks and I can crawl sites like amazon, google and facebook without problems by just calling their API.
With node I can do all my work async and with low memory footprint using simple http.get calls and some logic.
So no framework, no tools, nothing other than those two things
Re: Ask HN: What are best tools for web scraping?
#107Earlier quoted context omitted.
I've used it for some larger scrapes (nothing at the scale you're talking about, but still sizeable) and scrapy has very tight integration with scrapinghub.com to handle all of the deployment issues (including worker uptime, result storage, rate-limiting, etc). Not affiliated with them in any way, just have had a good experience using them in the past.
Every `hosted/cloud/saas/paas` goes into bazillions $$$ for anything largescale. Starting from aws bandwidth and including nearly every service on this earth.
I would hazard a guess that nearly all large scale use cases are negotiating those prices down quite a bit.
Re: Ask HN: What are best tools for web scraping?
#108Been getting blocked by recaptcha more and more, do any of these tools handle dealing with that or workarounds by default? Tried routing through proxies and swapping IP addresses, slowing down, etc... Any specific ways people get around that?
Re: Ask HN: What are best tools for web scraping?
#109So in general what do most people use web scraping for? Is it building up their on database of things not available via an API or something? It always sounds interesting, but the need for it is what confuses me.
Re: Ask HN: What are best tools for web scraping?
#110Been getting blocked by recaptcha more and more, do any of these tools handle dealing with that or workarounds by default? Tried routing through proxies and swapping IP addresses, slowing down, etc... Any specific ways people get around that?
You can use services like Anti-captcha [1]
We have a public API on Apify for that [2]
[1] https://anti-captcha.com/mainpage
[2] https://www.apify.com/petr_cermak/anti-captcha-recaptcha