Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

191–200 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#191

Post by dataslap »

ddorian43 wrote: post #66
sharmi wrote: post #63

Earlier quoted context omitted.

Yes. It beats building up your own crawler that handles all the edge cases. That said, before you reach the limits of scrapy, you will more likely be restricted by preventive measures put in place by twitter(or any other large website) to limit any one user hogging too much resources. Services like cloudflare or similar are aware of all the usual proxy servers and such and will immediately block such requests.

So how to do it ? You have to become google/bing ?

some "ethical" measures may do the trick to. scrapy has a setting to integrate delays + you can use fake headers. Some sites are pretty persistent with their cookies (include cookies in requests). It's all case by case basis

Re: Ask HN: What are best tools for web scraping?

#192

Post by pteredactyl »

marvinpinto wrote: post #15

I would recommend using Headless Chrome along with a library like puppeteer[0]. You get the advantage of using a real browser with which you run pages' javascript, load custom extensions, etc. [0]: https://github.com/GoogleChrome/puppeteer

I second this. I built using beautiful soup before and found Puppeteer much easier when interacting with the web. Especially nasty .NET sites.

Re: Ask HN: What are best tools for web scraping?

#195

Post by bdcravens »

CGamesPlay wrote: post #117
frik wrote: post #105

Earlier quoted context omitted.

> We use Redis to send task (update / discovery) to our crawlers. Some kind of queue implemented with Redis? How does it work?

Probably not what the GP uses, but Resque does this in Ruby land.

Sidekiq has emerged as a better option to Resque

Re: Ask HN: What are best tools for web scraping?

#196

Post by mapster »

samtc wrote: post #67

I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service. All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/ / / ) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception. We have cronjob (cron + jenkins) that trigger dataset upda…

Mind if I ask what info/data you are scraping and for what ends?

Re: Ask HN: What are best tools for web scraping?

#197

Post by PaulHoule »

stoneridge wrote: post #58
sharmi wrote: post #4

If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…

Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash. [0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-... [1] https://splash.readthedocs.io/en/stable/index.html [2] https://github.com/scrapy-plugins/scrapy-splash

HTMLUnit in Java is a good browser emulator and can be used to work JavaScript-heavy web sites, form submission, etc.

Re: Ask HN: What are best tools for web scraping?

#198

Post by bdcravens »

elchief wrote: post #157

Anyone who suggests a tool that can't understand JavaScript doesn't know what they are talking about You should be using Headless Chrome or Headless Firefox with a library that can control them in a user-friendly manner

I haven't dug deep recently, but if you need to automate browser download dialog this wasn't possible with Headless Chrome. (I'd love to find out that this has changed, and you can control it as well as you can with Selenium)

Re: Ask HN: What are best tools for web scraping?

#199

Post by wtfdaemon »

selllikesybok wrote: post #18
riekus wrote: post #2

Depends on your skillset and the data you want to scrape. I am testing waters for a new business that relies on scraped data. As a non programmer I had good success testing stuff with contentgrabber. Import.io also get mentioned a lot. Tried out octoparse but wast stable with the scraping.

I find the desktop tool by import.io a little challenging to work with. Their toy web-demo is solid for simple table extraction, though.

It's gotten light-years better since the desktop tool existed.

They've completely deprecated/sun-setted the desktop tool in favor of a greatly improved web application.

Re: Ask HN: What are best tools for web scraping?

#200

Post by setr »

ddorian43 wrote: post #66
sharmi wrote: post #63

Earlier quoted context omitted.

Yes. It beats building up your own crawler that handles all the edge cases. That said, before you reach the limits of scrapy, you will more likely be restricted by preventive measures put in place by twitter(or any other large website) to limit any one user hogging too much resources. Services like cloudflare or similar are aware of all the usual proxy servers and such and will immediately block such requests.

So how to do it ? You have to become google/bing ?

I had just spawned like 20 servers for a couple days on aws, but that was for a one-off scrape of some 4 million pages.
Post reply on HN