Ask HN: Any data scraping project ideas you can share?

#1

Post by dchuk » Wed, Nov 15, 2017, 10:55 PM UTC

I've done a good amount of scraping over the years, but haven't done much recently. Getting an itch to do some side projects in this area as well, so interested if anyone has a need for data that they can't currently get, or can't get in a clean structured way.

One example I've thought of recently (because of my own 9-5 job needs) is to scrape all the heavy duty truck company's sites and expose make model data (and images) via an API paired with a VIN decoder. Each OEM obfuscates their vehicle data in one way or another (JS widgets, only in PDFs, etc) and as far as I can tell, there aren't any API-based data sources for heavy duty/commercial vehicles.

Any other ideas?

Re: Ask HN: Any data scraping project ideas you can share?

#2

Post by joshribakoff » Thu, Nov 16, 2017, 2:44 AM UTC

Its a huge undertaking. You'd be competing with ACES/PIES, those guys charge about $10k a year last I checked. So there's definitely room to undercut them, if you can somehow get all that data.

Re: Ask HN: Any data scraping project ideas you can share?

#3

Post by dchuk » Thu, Nov 16, 2017, 2:59 AM UTC

joshribakoff wrote: post #2

Its a huge undertaking. You'd be competing with ACES/PIES, those guys charge about $10k a year last I checked. So there's definitely room to undercut them, if you can somehow get all that data.

Thanks for the reply. I was thinking of just using the scraped data to create an API that can take a VIN and give you the specs of that heavy duty vehicle (engine, weight, body type options, etc). I couldn't even imagine how crazy it would be to try and collect all of the parts data for every vehicle.

Re: Ask HN: Any data scraping project ideas you can share?

#4

Post by dchuk » Thu, Nov 16, 2017, 3:05 AM UTC

Here's another idea I just thought of randomly:

Scrape and monitor a company's competitor's job listings for them. Some of that data might be difficult to get given the nature of job sites and craigslist and such, but could be interesting to accumulate all of that (including from the company's own site) so you can get an idea when they are hiring.

Maybe.

Re: Ask HN: Any data scraping project ideas you can share?

#5

Post by staticautomatic » Thu, Nov 16, 2017, 3:19 AM UTC

Aggregate data on weather and soil composition to identify areas of land around the world most similar to famous wine producing regions.

Re: Ask HN: Any data scraping project ideas you can share?

#6

Post by uptownfunk » Thu, Nov 16, 2017, 4:11 AM UTC

Good quality, accurate, high resolution stock / options / futures data

Re: Ask HN: Any data scraping project ideas you can share?

#7

Post by richardknop » Thu, Nov 16, 2017, 5:41 AM UTC

Well the obvious scraping business ideas would be hotel rooms and airplane tickets. You might think this area is already saturated but I think there’s still room for disruption and you can capture niche giants like Expedia / Skyscanner / Agoda / Booking don’t handle well. Or you could do b2b with these companies.

Also, what about scraping restaurant menus and offering a food search engine?

Re: Ask HN: Any data scraping project ideas you can share?

#8

Post by gyvastis » Thu, Nov 16, 2017, 7:53 AM UTC

dchuk wrote: post #4

Here's another idea I just thought of randomly: Scrape and monitor a company's competitor's job listings for them. Some of that data might be difficult to get given the nature of job sites and craigslist and such, but could be interesting to accumulate all of that (including from the company's own site) so you can get an idea when they are hiring. Maybe.

Sorry if I've missed the main idea behind this, but why would that be relevant?

Re: Ask HN: Any data scraping project ideas you can share?

#9

Post by gyvastis » Thu, Nov 16, 2017, 7:54 AM UTC

staticautomatic wrote: post #5

Aggregate data on weather and soil composition to identify areas of land around the world most similar to famous wine producing regions.

Wow! That's an awesome idea.

Re: Ask HN: Any data scraping project ideas you can share?

#10

Post by gandutraveler » Thu, Nov 16, 2017, 8:11 AM UTC

I have been trying to scrape travel places and recommendations data from TripAdvisor and other travel sites. Read about scrapy in another hn post yesterday and have been trying to get it running. Would appreciate any help on this.

sam.xenai [at] gmail [dot] com