Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#231

Post by Launchr »

samtc wrote: post #158
CGamesPlay wrote: post #118

Earlier quoted context omitted.

I have a similar set up! How do you monitor for failures and deal with the scrape target changing?

We monitor exceptions with Sentry. We store raw data so we don't have to hurry to fix the ETL, we only have to fix navigation logic and we keep crawling.

Sorry if it's a stupid question/example/comparison, just trying to understand better: You're storing the full html data instead of reaching into the specific div's for the data you might need? This way, separating the fetching from the parsing?

I'm a scraping rookie, and I usually fetch + parse in the same call, this might resolve some issues for me :) thanks!

Re: Ask HN: What are best tools for web scraping?

#234

Post by jimsmart »

Launchr wrote: post #231
samtc wrote: post #158

Earlier quoted context omitted.

We monitor exceptions with Sentry. We store raw data so we don't have to hurry to fix the ETL, we only have to fix navigation logic and we keep crawling.

Sorry if it's a stupid question/example/comparison, just trying to understand better: You're storing the full html data instead of reaching into the specific div's for the data you might need? This way, separating the fetching from the parsing? I'm a scraping rookie, and I usually fetch + parse in the same call, this might resolve some issues for me :) thanks!

When I've done scraping, I've always taken this approach also: I decouple my process into paired fetch-to-local-cache-folder and process-cached-files stages.

I find this useful for several reasons, but particularly if you want to recrawl the same site for new/updated content, or if you decide to grab extra data from the pages (or, indeed, if your original parsing goes wrong or meets pages it wasn't designed for).

Related: As well as any pages I cache, I generally also have each stage output a CSV (requested url, local file name, status, any other relevant data or metadata), which can be used to drive later stages, or may contain the final output data.

Requesting all of the pages is the biggest time sink when scraping — it's good to avoid having to do any portion of that again, if possible.

Re: Ask HN: What are best tools for web scraping?

#235

Post by hydragit »

danso wrote: post #102

Always fascinated by how diverse the discussion and answers is for HN threads on web-scraping. Goes to show that "web-scraping" has a ton of connotations, everything from automated-fetching of URLs via wget or cURL, to data management via something like scrapy. Scrapy is a whole framework that may be worthwhile, but if I were just starting out for a specific task, I would use: - requests http://docs.python-requests.o…

> Python 3, AFAIK, doesn't have anything as handy as Ruby/Perl's Mechanize. But using the web developer tools you can usually figure out the requests made by the browser and then use the Session object in the Requests library to deal with stateful requests

You could also use the WebOOB (http://weboob.org) framework. It's built on requests+lxml and it provides a Browser class usable like mechanize's one (ability to access doc, select HTML forms, etc.).

It also has nice companion features like associating url patterns to some custom Page classes where you can write what data to retrieve when a page with this url pattern is browsed.

Re: Ask HN: What are best tools for web scraping?

#237

Post by vrathee »

If you are looking for SaaS or managed services, Try https://www.agenty.com/

Agenty is cloud-hosted web scraping app and you can setup scraping agents using their point and click CSS Selector Chrome extension to extract anything from HTML with these 3 modes below: - TEXT : Simple clean text - HTML : Outer or Inner HTML - ATTR : Any attribute of a html tag like image src, hyperlink href…

Or advance mode like REGEX, XPATH etc.

And then save the scraping agent to execute on cloud-hosted app with most advanced features like batch crawling, scheduling, multiple website scraping simultaneously without worrying in ip-address block or speed like never before.

Re: Ask HN: What are best tools for web scraping?

#238

Post by jacinda »

If you're specifically looking at news articles, go for the Python library Newspaper: http://newspaper.readthedocs.io/en/latest/

Auto-detection of languages, and will automatically give you things like the following:

>>> article.parse()

>>> article.authors [u'Leigh Ann Caldwell', 'John Honway']

>>> article.text u'Washington (CNN) -- Not everyone subscribes to a New Year's resolution...'

>>> article.top_image u'http://someCDN.com/blah/blah/blah/file.png'

>>> article.movies [u'http://youtube.com/path/to/link.com', ...]

Re: Ask HN: What are best tools for web scraping?

#240

Post by nn757 »

phsource wrote: post #87

For someone on a Javascript stack, I highly recommend combining a requester (e.g., "request" or "axios") with Cheerio, a server-side jQuery clone. Having a familiar, well-known interface for selection helps a lot. We use this stack at WrapAPI ( https://wrapapi.com ), which we highly recommend as a tool to turn webpages into APIs. It doesn't completely do all the scraping (you still need to write a script), but it doe…

Isn't cheerio only for static content?
Post reply on HN