Scraping Hacker News on a Schedule with TaskPipes • Hacker News Board

Re: Scraping Hacker News on a Schedule with TaskPipes

#11

Post by chdir » Wed, Sep 23, 2015, 2:20 PM UTC

dhruvkar wrote: post #10

chdir wrote: post #9

I've seen something similar more than once on HN but my google-fu is failing. What other similar services are out there that'll let you smartly scrape a page into a spreadsheet. In the past, I've tried Google spreadsheets with "ImportXML" option but got frustrated after a bit and resorted to python.

Kimonolabs.com Allows you to scrape then import that data into a google spreadsheet. But its not the only way to consume the data.

Thanks, that's the one I had in mind. Also found : import.io, scrapinghub.com & 80legs.com

Re: Scraping Hacker News on a Schedule with TaskPipes

#12

Post by tynman » Wed, Sep 23, 2015, 6:57 PM UTC

What Open Source projects are similar? It seems like data transformation tasks like this are a universal problem. I'd just do map/reduce, but that's the advantage that comes with being a developer. Great tools for non-developers to do limited programming tasks are incredibly useful.

Re: Scraping Hacker News on a Schedule with TaskPipes

#13

Post by juanescobarcom » Thu, Sep 24, 2015, 5:09 PM UTC

Check out Diffbot @ http://www.diffbot.com/ -- you can setup repeat crawls and extract data into .csv/Excel file format or JSON with Crawlbot API, extract data automatically with automatic APIs, or Custom API Toolkit. You can get a free trial account at https://www.diffbot.com/plans/trial to try it out, and you can get paid plans at http://www.diffbot.com/pricing/.