Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#202

Post by sp0rk »

elchief wrote: post #157

Anyone who suggests a tool that can't understand JavaScript doesn't know what they are talking about You should be using Headless Chrome or Headless Firefox with a library that can control them in a user-friendly manner

There are a great many sites that degrade gracefully when JS support is not available. It makes absolutely no sense to waste the resources required to run a full headless browser when simple HTTP requests will retrieve the same information faster, more efficiently, and in a way that's easier to parallelize.

Re: Ask HN: What are best tools for web scraping?

#203

Post by grad_ml »

ajimix wrote: post #156
grad_ml wrote: post #155

Earlier quoted context omitted.

To me, it seems proxycrawl is very expensive! If I may ask, can you talk little about your crawl volume and cost?

I’m crawling around 80-120M per month and the price for me fits my needs. But I suggest that you contact them if you have special needs or requirements. Also you have to consider the amount of work, time and money that you will save by not maintaining your own system to avoid blocks and bans from the websites you are trying to crawl. With them you just call an API endpoint and you don't have to care about all that

Thanks. One follow up questions, in JS heavy pages, how operations like infinite scrolling etc are exposed/executed?

Re: Ask HN: What are best tools for web scraping?

#204

Post by xur17 »

sp0rk wrote: post #202
elchief wrote: post #157

Anyone who suggests a tool that can't understand JavaScript doesn't know what they are talking about You should be using Headless Chrome or Headless Firefox with a library that can control them in a user-friendly manner

There are a great many sites that degrade gracefully when JS support is not available. It makes absolutely no sense to waste the resources required to run a full headless browser when simple HTTP requests will retrieve the same information faster, more efficiently, and in a way that's easier to parallelize.

A lot of times you can also watch the api calls JS pages (or apps) make and retrieve nice structured json data.

I personally avoid executing js unless it's necessary, as it adds more complexity, and is noticeably more brittle.

Re: Ask HN: What are best tools for web scraping?

#207

Post by btb »

We have been using kapow robosuite for close to 10 years now. Its a commercial GUI based tool which have worked well for us, it saves us a lot of maintenance time compared to our previous hand-rolled code extraction pipeline. Only problem is that its very expensive(pricing seems catered towards very large enterprises).

So I was really hoping this this thread would have revealed some newer commercial GUI-based alternatives(on-premise, not SaaS). Because I dont really ever want to go back the maintenance hell of hand rolled robots ever again :)

Re: Ask HN: What are best tools for web scraping?

#208

Post by imjasonmiller »

ravenstine wrote: post #166

It depends on what you're trying to do. For most things, I use Node.js with the Cheerio library, which is basically a stripped-down version of jQuery without the need for a browser environment. I find using the jQuery API far more desirable than the clunky, hideous Beautiful Soup or Nokogiri APIs. For something that requires an actual DOM or code execution, PhantomJS with Horseman works well, though everyone is talki…

I have been playing around with Cheerio for a short while and it is quite cool! Although extracting comments wasn't as straightforward as I thought it would be.

Do you have any experience with processing and scraping large files using Cheerio? It doesn't support streaming does it? I am currently faced with processing a ~75 MB XML and I am not sure if Cheerio is suited for that.

Post reply on HN