For a little diversity on tools, if you're looking for something quick that others can access the data easily - Google Apps script in a Google Sheet can be quite useful.
https://sites.google.com/site/scriptsexamples/learn-by-examp...
Ask HN: What are best tools for web scraping?
181–190 of 243 posts
Re: Ask HN: What are best tools for web scraping?
#182Earlier quoted context omitted.
Phantom is woefully out of date, you need a polyfill even for Function.bind. Firefox dropped support for Selenium in 47, and chromedriver only supports it with a wrapper called chromedriver.
Are you talking about Selenium WebDriver or Selenium IDE (the record/playback tool for Firefox)? Those are two separate things. Selenium WebDriver implements is a cross-browser W3C-standard and Firefox very much still supports it.
Hmm, I guess through geckodriver, which is a parallel to chromedriver? Just reading through https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionet... which starts with a warning about "rough edges" and "substantial differences".
Re: Ask HN: What are best tools for web scraping?
#183i did a quick search and didnt see this listed here:
Re: Ask HN: What are best tools for web scraping?
#184One of the important avenues to scrape AJAX heavy and phantomjs avoiding websites is using the google chrome extension support. They can mirror the dom and send it to an external server for processing where we can use python lxml to xpath to appropriate nodes. This worked for me to scrape Google, before we hit the capatcha. If anyone is interested, i can share code i wrote to scrape websites ! If you can scrape findt…
> This worked for me to scrape Google, before we hit the capatcha. If Google wanted to give back something to the community, it would offer cheap automated searches (current prices are absurd). Another thing - more depth after the first 1000 results. Sometimes you want to know the next result. We shouldn't need to do all these stupid things to batch query a search engine, it should be open. That makes it all the more…
I absolutely agree, and I am thinking strategies to even automate the capatcha, using crowdsourcing or better, using AI/ML ( which is not trivial ).
duckduckgo is good but not there yet.
Would you be interested to work on a search engine ? Some projects are bitfunnel and so forth.
Re: Ask HN: What are best tools for web scraping?
#185If you can get away without a JS environment, do so. Something like scrapy will be much easier than a full browser environment. If you cannot, don’t bother going halfway and just go straight for headless chrome or Firefox. Unfortunately Selenium seems to be past its useful life as Firefox dropped support and chrome has a chrome driver which wraps around it. Phantom.js is woefully out of date and since it’s a differen…
I manage the WebDriver work at Mozilla making Firefox work with Selenium. I can categorically State we haven’t killed Selenium. We, over the last few years, have invested more in Selenium than other browsers. Selenium IDE no longer works in Firefox for a number of reasons; 1) Selenium IDE didn’t have a maintainer 2) Selenium IDE is a Firefox add on and Mozilla changed how adding worked. They did this for numerous sec…
My apologies, I was mistaken, but I can't edit my post now. It looks like the selenium code has moved into something called geckodriver, which I suppose is a wrapper around the underlying Marionette protocol.
Re: Ask HN: What are best tools for web scraping?
#186The absolute best tool i have found for scraping is Visual Web Ripper.
It is not open source, and runs in windows only, but it is one of the easiest to use tools that i have found. I can set up scrapes entirely visually, and it handles complex cases like infinite scroll pages, highly javascript dependent pages and the like. I really wish there were an open source solution that was as good as this one.
I use it with one of my clients professionally. Their support is VERY good btw.
Re: Ask HN: What are best tools for web scraping?
#187If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…
I've recently made a little project with scrapy (for crawling) and BeautifulSoup (for parsing html) and it works out great. One more thing to add to the above list are pipelines, they make downloading files quite easy.
scrapy has a pretty decent parser too
Re: Ask HN: What are best tools for web scraping?
#188Re: Ask HN: What are best tools for web scraping?
#189If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…
Would you still recommend Scrapy if the task wasn't specifically crawling?
depends on the task. For example they have a decent file/image downloading middleware.
Re: Ask HN: What are best tools for web scraping?
#190If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…
I've had mixed results with scrapy, probably more based in my inexperience than other thing, but for example retrieving a posting in idealista.com with vanilla scrapy begets an error page whereas a basic wget command retrieves the correct page.
So the learning curve for simple things makes me jump to bash scripts; scrapy might prove more valuable when your project starts to scale.
But also of course: normally the best tool is the one you already know!