Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets? • Hacker News Board

Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#1

Post by kisamoto » Fri, Sep 11, 2020, 8:48 AM UTC

I'm fairly familiar with web scraping/crawling however I was wondering if there is a company/tool that has re-usable modules for scraping common websites?

Examples could include: scraping article texts from news websites; extracting recipes from Good Food etc.

Rather than rewriting what others have - is there an existing library of these scrapers/crawlers to use 'out of the box'?

Re: Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#2

Post by thedevindevops » Fri, Sep 11, 2020, 10:23 AM UTC

What you're looking for sounds very open ended but the closest thing I can think of is the Huginn project on github?

Re: Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#3

Post by kilroy123 » Fri, Sep 11, 2020, 10:58 AM UTC

This is sorely needed IMO.

Re: Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#4

Post by atmosx » Fri, Sep 11, 2020, 3:05 PM UTC

Try: https://www.scrapinghub.com/

Re: Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#5

Post by mendelevium » Fri, Sep 11, 2020, 3:23 PM UTC

For extracting news articles: https://newspaper.readthedocs.io/en/latest/

Re: Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#6

Post by kisamoto » Fri, Sep 11, 2020, 3:27 PM UTC

mendelevium wrote: post #5

For extracting news articles: https://newspaper.readthedocs.io/en/latest/

While not general purpose templates, for news articles that seems exceptionally useful

Re: Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#7

Post by kisamoto » Fri, Sep 11, 2020, 3:28 PM UTC

atmosx wrote: post #4

Try: https://www.scrapinghub.com/

I know ScrapingHub actually, has some useful tools but as far as I know they don't have a library of off-the-shelf scrapers for popular websites?

Re: Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets?

#8

Post by abarrettwilsdon » Sat, Sep 12, 2020, 6:19 PM UTC

Not exactly what you're looking for, but there's a OSS Chrome Extension that allows you to record your actions in browser and transcribes them into Nightmare.js code:

https://github.com/segmentio/daydream

Probably the best you're going to get - most things worth scraping are worth money, and as such are not freely available