Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#171

Post by j_s »

Bromskloss wrote: post #39
jackschultz wrote: post #16

I've actually wrote about this! General tips that I've found from doing more than a few projects [0], and then an overview of Python libraries I use [1]. If you don't want to clock on the links, requests and BeautifulSoup / lxml is all you need 90% of the time. Throw gevent in there and you can get a lot of scraping done in not as much time as you think it would take. And as long as we're talking about web scraping,…

> BeautifulSoup / lxml When should one use one or the other, would you say?

Use https://github.com/kovidgoyal/html5-parser, which (in my limited understanding) does a better job faster and is backwards-compatible with both.

Recommendation by the author (of Calibre fame) on a similar discussion: https://news.ycombinator.com/item?id=15539853

Dedicated discussion: https://news.ycombinator.com/item?id=14588333

Re: Ask HN: What are best tools for web scraping?

#172

Post by visarga »

ajimix wrote: post #106

I've been doing scraping for many years and at the end it's always the same, you build a lot of stuff to bypass site restrictions and finally, once you are done, you can start scraping. It all goes fine until the site you are scrapping bans you... So what I do now? - proxycrawl https://proxycrawl.com - node http://nodejs.org With proxycrawl I don't need to worry about bans or blocks and I can crawl sites like amazon,…

Sounds like an advert for an expensive product (proxycrawl)

Re: Ask HN: What are best tools for web scraping?

#173

Post by visarga »

bootcat wrote: post #154

One of the important avenues to scrape AJAX heavy and phantomjs avoiding websites is using the google chrome extension support. They can mirror the dom and send it to an external server for processing where we can use python lxml to xpath to appropriate nodes. This worked for me to scrape Google, before we hit the capatcha. If anyone is interested, i can share code i wrote to scrape websites ! If you can scrape findt…

> This worked for me to scrape Google, before we hit the capatcha.

If Google wanted to give back something to the community, it would offer cheap automated searches (current prices are absurd). Another thing - more depth after the first 1000 results. Sometimes you want to know the next result. We shouldn't need to do all these stupid things to batch query a search engine, it should be open. That makes it all the more important to invent an open-source, federated search engine, so we can query to our heart's content (and have privacy).

Re: Ask HN: What are best tools for web scraping?

#174

Post by AznHisoka »

austincheney wrote: post #169
AznHisoka wrote: post #167

Earlier quoted context omitted.

but how do you use that code? its javascript, right? how would you use it if your crawler is written in Ruby or Python?

You could write a crawler in any language. Crawling is easy as you are listening for HTTP traffic and analyzing the HTML in the response. To accurately get the content in dynamically executed pages you need to interact with the DOM. This is the reason Google updated its crawler to execute JavaScript.

Yep, I know, but that means if I am writing the crawler in Ruby/Python, this is not something I can do, right?

Re: Ask HN: What are best tools for web scraping?

#175

Post by pudo »

We're about to announce a new Python scraping toolkit, memorious: https://github.com/alephdata/memorious - it's a pretty lightweight toolkit, using YAML config files to glue together pre-built and custom-made components into flexible and distributed pipelines. A simple web UI helps track errors and execution can be scheduled via celery.

We looked at scrapy, but it just seemed like the wrong type of framing for the type of scrapers we build: requests, some html/xml parser, and output into a service API or a SQL store.

Maybe some people will enjoy it.

Re: Ask HN: What are best tools for web scraping?

#176

Post by colinchartier »

We had a really tough time scraping dynamic web content using scrapy, and both scrapy and selenium require you to write a program (and maintain it) for every separate website that you have to scrape. If the website's structure changes you need to debug your scraper. Not fun if you need to manage more than 5 scrapers.

It was so hard that we made our own company JUST to scrape stuff easily without requiring programming. Take a look at https://www.parsehub.com

Re: Ask HN: What are best tools for web scraping?

#178

Post by austincheney »

AznHisoka wrote: post #174
austincheney wrote: post #169

Earlier quoted context omitted.

You could write a crawler in any language. Crawling is easy as you are listening for HTTP traffic and analyzing the HTML in the response. To accurately get the content in dynamically executed pages you need to interact with the DOM. This is the reason Google updated its crawler to execute JavaScript.

Yep, I know, but that means if I am writing the crawler in Ruby/Python, this is not something I can do, right?

Yes. The crawler can be written in nearly any language. The actual scraper probably has to be written in JavaScript in order to access and interact with the DOM as the user would and thereby gain access to content that is not present by default.

Re: Ask HN: What are best tools for web scraping?

#180

Post by sebcat »

danso wrote: post #137
sebcat wrote: post #130

Earlier quoted context omitted.

lxml can be hit-or-miss on HTML5 docs. I've had greater success with a modified version of gumbo-parser.

Ah very cool, had seen various python libraries about HTML5, but not gumbo (or at least I had starred it). https://github.com/google/gumbo-parser Is the modified version you use a personal version or a well-known fork?

> Is the modified version you use a personal version or a well-known fork?

I had a specific thing I needed to do, gumbo-parser was a good match, I poked at it a little and moved on. It started with this[1] commit, then I did some other work locally which was not pushed because google/gumbo-parser is without an owner/maintainer. There are a couple of forks, but no/little adoption it seems.

[1] https://github.com/sebcat/gumbo-parser/commit/c158f8090c2df0...

Post reply on HN