111–120 of 243 posts
scapy is fine but selenium, phantom, etc are all outdated IMO
Headless chrome in the form of puppeteer ( https://github.com/GoogleChrome/puppeteer ) or Chromeless ( https://github.com/graphcool/chromeless ) or for smaller gigs use nightmare.js ( http://www.nightmarejs.org/ ). scapy is fine but selenium, phantom, etc are all outdated IMO
For what reason? Genuine question.
Always fascinated by how diverse the discussion and answers is for HN threads on web-scraping. Goes to show that "web-scraping" has a ton of connotations, everything from automated-fetching of URLs via wget or cURL, to data management via something like scrapy. Scrapy is a whole framework that may be worthwhile, but if I were just starting out for a specific task, I would use: - requests http://docs.python-requests.o…
Headless chrome in the form of puppeteer ( https://github.com/GoogleChrome/puppeteer ) or Chromeless ( https://github.com/graphcool/chromeless ) or for smaller gigs use nightmare.js ( http://www.nightmarejs.org/ ). scapy is fine but selenium, phantom, etc are all outdated IMO
> are all outdated IMO For what reason? Genuine question.
I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service. All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/ / / ) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception. We have cronjob (cron + jenkins) that trigger dataset upda…
> We use Redis to send task (update / discovery) to our crawlers. Some kind of queue implemented with Redis? How does it work?
I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service. All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/ / / ) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception. We have cronjob (cron + jenkins) that trigger dataset upda…
Been getting blocked by recaptcha more and more, do any of these tools handle dealing with that or workarounds by default? Tried routing through proxies and swapping IP addresses, slowing down, etc... Any specific ways people get around that?
If you prefer an API as a service that can pre-render pages, I built Page.REST (https://www.page.rest). It allows you to get rendered page content via CSS selectors as a JSON response.