Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#62

Post by austincheney »

This is perhaps the fastest way to screenscrape a dynamically executed website.

1. First go get and run this code, which allows immediate gathering of all text nodes from the DOM: https://github.com/prettydiff/getNodesByType/blob/master/get...

2. Extract the text content from the text nodes and ignore nodes that contain only white space:

let text = document.getNodesByType(3), a = 0, b = text.length, output = []; do { if ((/^(\s+)$/).test(text[a].textContent) === false) { output.push(text[a].textContent); } a = a + 1; } while (a That will gather ALL text from the page. Since you are working from the DOM directly you can filter your results by various contextual and stylistic factors. Since this code is small and executes stupid fast it can be executed by bots easily.

Test this out in your browser console.

Re: Ask HN: What are best tools for web scraping?

#63

Post by sharmi »

ddorian43 wrote: post #49
sharmi wrote: post #4

If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…

Would you recommend it for scalable projects ? Like, crawl twitter or tumblr ?

Yes. It beats building up your own crawler that handles all the edge cases. That said, before you reach the limits of scrapy, you will more likely be restricted by preventive measures put in place by twitter(or any other large website) to limit any one user hogging too much resources. Services like cloudflare or similar are aware of all the usual proxy servers and such and will immediately block such requests.

Re: Ask HN: What are best tools for web scraping?

#65

Post by DrSayre »

Doctor_Fegg wrote: post #11

If you speak Ruby, mechanize is good: https://github.com/sparklemotion/mechanize

I generally use mechanize when I need to scrape something from the web. I found this awhile back and it's helped me https://www.chrismytton.uk/2015/01/19/web-scraping-with-ruby...

Re: Ask HN: What are best tools for web scraping?

#66

Post by ddorian43 »

sharmi wrote: post #63
ddorian43 wrote: post #49

Earlier quoted context omitted.

Would you recommend it for scalable projects ? Like, crawl twitter or tumblr ?

Yes. It beats building up your own crawler that handles all the edge cases. That said, before you reach the limits of scrapy, you will more likely be restricted by preventive measures put in place by twitter(or any other large website) to limit any one user hogging too much resources. Services like cloudflare or similar are aware of all the usual proxy servers and such and will immediately block such requests.

So how to do it ? You have to become google/bing ?

Re: Ask HN: What are best tools for web scraping?

#67

Post by samtc »

I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service.

All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to///) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception.

We have cronjob (cron + jenkins) that trigger dataset update and discovery. For example, we scrape corporate registry, so everyday we update the 20k oldest companies version. We also implement "discovery" logic in all of our crawlers so they can find new data (ex.: newly registered company). We use Redis to send task (update / discovery) to our crawlers.

Post reply on HN