If you need simple scraping, I like traditional http request lib. For more robust scraping (ie clicking buttons / filling text), use capybara and either phantomjs or chromedriver - easy to install using homebrew!
Ask HN: What are best tools for web scraping?
161–170 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#162Earlier quoted context omitted.
> are all outdated IMO For what reason? Genuine question.
Phantom is woefully out of date, you need a polyfill even for Function.bind. Firefox dropped support for Selenium in 47, and chromedriver only supports it with a wrapper called chromedriver.
Are you talking about Selenium WebDriver or Selenium IDE (the record/playback tool for Firefox)? Those are two separate things. Selenium WebDriver implements is a cross-browser W3C-standard and Firefox very much still supports it.
Re: Ask HN: What are best tools for web scraping?
#163If you can get away without a JS environment, do so. Something like scrapy will be much easier than a full browser environment. If you cannot, don’t bother going halfway and just go straight for headless chrome or Firefox. Unfortunately Selenium seems to be past its useful life as Firefox dropped support and chrome has a chrome driver which wraps around it. Phantom.js is woefully out of date and since it’s a differen…
Firefox did not drop support for Selenium. Selenium IDE, a record/playback test creation tool, stopped working in newer versions of Firefox, but a) Selenium IDE is only one part of the Selenium project, and b) The Selenium team is working on a new version of IDE compatible with the new Firefox add-on APIs.
Re: Ask HN: What are best tools for web scraping?
#164Earlier quoted context omitted.
> BeautifulSoup / lxml When should one use one or the other, would you say?
You can use the BeautifulSoup API with the `lxml` parser: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#insta... I've heard that `lxml` can choke on certain badly-formed markup, but it's very fast. Personally has never failed on me.
LXML also is known to have memory leaks [0][1], so be careful using it in any kind of automated system that will be parsing lots of small documents. I personally encountered this issue, and actually caused to abandon a project until months later when I found the references I linked above. It works nice and fast for one-off tasks, though.
Also, a question: how often do you really encounter badly-formed markup in the wild? How hard is it really to get HTML right? It seems pretty simple, just close tags and don't embed too much crazy stuff in CDATA. Yet I often read about how HTML parsers must be "permissive" while XML parsers don't need to be. I've never had a problem parsing bad markup; usually my issues have to do with text encoding (either being mangled directly or being correctly-encoded vestiges of a prior mangling) and the other usual problems associated with text data.
[0]: https://benbernardblog.com/tracking-down-a-freaky-python-mem...
Re: Ask HN: What are best tools for web scraping?
#165I made a crawler https://github.com/jahaynes/crawler
It outputs to the warc file format (https://en.wikipedia.org/wiki/Web_ARChive), in case your workflow is to gather web pages and then process them afterwards.
Re: Ask HN: What are best tools for web scraping?
#166It depends on what you're trying to do.
For most things, I use Node.js with the Cheerio library, which is basically a stripped-down version of jQuery without the need for a browser environment. I find using the jQuery API far more desirable than the clunky, hideous Beautiful Soup or Nokogiri APIs.
For something that requires an actual DOM or code execution, PhantomJS with Horseman works well, though everyone is talking about headless Chrome these days so IDK. I've not had nearly as many bad experiences with PhantomJS as others have purportedly experienced.
Re: Ask HN: What are best tools for web scraping?
#167Earlier quoted context omitted.
And how do you do #1? Node, I presume?
No, manually go there and copy/paste the code. Then when building your scraper bot use that code.
but how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?
Re: Ask HN: What are best tools for web scraping?
#168I use a combination of Selenium and python packages (beautifulsoup). I'm primarily interested in scraping data that is supplied via javascript, and I find Selenium to be the most reliable way scrape that info. I use BS when the scraped page has a lot of data, thereby slowing down Selenium, and I pipe the page source from Selenium, with all javascript rendered, into BS. I use explicit waits exclusively (no direct call…
>I'm primarily interested in scraping data that is supplied via javascript, and I find Selenium to be the most reliable way scrape that info. Have you found that you aren't able to find accessible APIs to request against? Have you ever tried to contact the administrators to see if there's an API you could access? Are you scraping data that would be against ToS if you tried to get it in a way that would benefit both y…
[deleted]
Re: Ask HN: What are best tools for web scraping?
#169Earlier quoted context omitted.
No, manually go there and copy/paste the code. Then when building your scraper bot use that code.
but how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?
You could write a crawler in any language. Crawling is easy as you are listening for HTTP traffic and analyzing the HTML in the response.
To accurately get the content in dynamically executed pages you need to interact with the DOM. This is the reason Google updated its crawler to execute JavaScript.
Re: Ask HN: What are best tools for web scraping?
#170If you can get away without a JS environment, do so. Something like scrapy will be much easier than a full browser environment. If you cannot, don’t bother going halfway and just go straight for headless chrome or Firefox. Unfortunately Selenium seems to be past its useful life as Firefox dropped support and chrome has a chrome driver which wraps around it. Phantom.js is woefully out of date and since it’s a differen…
I manage the WebDriver work at Mozilla making Firefox work with Selenium. I can categorically State we haven’t killed Selenium. We, over the last few years, have invested more in Selenium than other browsers.
Selenium IDE no longer works in Firefox for a number of reasons; 1) Selenium IDE didn’t have a maintainer 2) Selenium IDE is a Firefox add on and Mozilla changed how adding worked. They did this for numerous security reasons.