If you know Java, then my go to library is Jsoup https://jsoup.org/ It lets you use jQuery-like selectors to extract data. Like this: Elements newsHeadlines = doc.select("#mp-itn b a");
+1 Saves a ton of time, and very simple to use
Ask HN: What are best tools for web scraping?
121–130 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#122I use a combination of Selenium and python packages (beautifulsoup). I'm primarily interested in scraping data that is supplied via javascript, and I find Selenium to be the most reliable way scrape that info. I use BS when the scraped page has a lot of data, thereby slowing down Selenium, and I pipe the page source from Selenium, with all javascript rendered, into BS.
I use explicit waits exclusively (no direct calls like `driver.find_foo_by_bar`), and find it vastly improves selenium reliability. (Shameless plug) I have a python package, Explicit[1], that makes it easier to use explicit waits.
Re: Ask HN: What are best tools for web scraping?
#123I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service. All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/ / / ) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception. We have cronjob (cron + jenkins) that trigger dataset upda…
> We use Redis to send task (update / discovery) to our crawlers. Some kind of queue implemented with Redis? How does it work?
See https://sidekiq.org for instance.
Re: Ask HN: What are best tools for web scraping?
#124I've been doing scraping for many years and at the end it's always the same, you build a lot of stuff to bypass site restrictions and finally, once you are done, you can start scraping. It all goes fine until the site you are scrapping bans you... So what I do now? - proxycrawl https://proxycrawl.com - node http://nodejs.org With proxycrawl I don't need to worry about bans or blocks and I can crawl sites like amazon,…
Proxycrawl seemed interesting, so I just tried it out. It appears to have problems with redirects, which is something I expect they would have figured out.
Re: Ask HN: What are best tools for web scraping?
#125If you can get away without a JS environment, do so. Something like scrapy will be much easier than a full browser environment. If you cannot, don’t bother going halfway and just go straight for headless chrome or Firefox. Unfortunately Selenium seems to be past its useful life as Firefox dropped support and chrome has a chrome driver which wraps around it. Phantom.js is woefully out of date and since it’s a differen…
Can you explain a little more? How do you drive FF/Chrome without Selenium?
Re: Ask HN: What are best tools for web scraping?
#126I use a combination of Selenium and python packages (beautifulsoup). I'm primarily interested in scraping data that is supplied via javascript, and I find Selenium to be the most reliable way scrape that info. I use BS when the scraped page has a lot of data, thereby slowing down Selenium, and I pipe the page source from Selenium, with all javascript rendered, into BS. I use explicit waits exclusively (no direct call…
>I'm primarily interested in scraping data that is supplied via javascript, and I find Selenium to be the most reliable way scrape that info.
Have you found that you aren't able to find accessible APIs to request against? Have you ever tried to contact the administrators to see if there's an API you could access? Are you scraping data that would be against ToS if you tried to get it in a way that would benefit both you and the target web site?
Re: Ask HN: What are best tools for web scraping?
#127I've been doing scraping for many years and at the end it's always the same, you build a lot of stuff to bypass site restrictions and finally, once you are done, you can start scraping. It all goes fine until the site you are scrapping bans you... So what I do now? - proxycrawl https://proxycrawl.com - node http://nodejs.org With proxycrawl I don't need to worry about bans or blocks and I can crawl sites like amazon,…
ProxyCrawl looks very interesting, I have already made 1000 free requests to Instagram. I will investigate more on it
Re: Ask HN: What are best tools for web scraping?
#128Whatever you end up using for scraping, I beg you to pick a unique user-agent which allows a webmaster to understand which crawler is it, to better allow it to pass through (or be banned, depending).
Don't stick with the default "scrapy" or "Ruby" or "Jakarta Commons-HttpClient/...", which end up (justly) being banned more easily than unique ones, like "ABC/2.0 - https://example.com/crawler" or the like.
Re: Ask HN: What are best tools for web scraping?
#129If you can get away without a JS environment, do so. Something like scrapy will be much easier than a full browser environment. If you cannot, don’t bother going halfway and just go straight for headless chrome or Firefox. Unfortunately Selenium seems to be past its useful life as Firefox dropped support and chrome has a chrome driver which wraps around it. Phantom.js is woefully out of date and since it’s a differen…
Phantom.js is not out of date. Research something before you talk about it. I use it consistently, and the GitHub repo is active. Open an issue if something isn't working for you. The last one was closed as recent as 9 days ago:
https://github.com/ariya/phantomjs/issues?q=is%3Aissue+is%3A...
Re: Ask HN: What are best tools for web scraping?
#130Always fascinated by how diverse the discussion and answers is for HN threads on web-scraping. Goes to show that "web-scraping" has a ton of connotations, everything from automated-fetching of URLs via wget or cURL, to data management via something like scrapy. Scrapy is a whole framework that may be worthwhile, but if I were just starting out for a specific task, I would use: - requests http://docs.python-requests.o…
lxml can be hit-or-miss on HTML5 docs. I've had greater success with a modified version of gumbo-parser.