Scrape websites like TripAdvisor, Amazon, etc. for the ratings and compute an actual rating not based on averages. I've seen a few articles on how ratings are shown on those websites recently and they never seem to actually reflect the truth.
Ask HN: Any data scraping project ideas you can share?
11–20 of 35 posts
Re: Ask HN: Any data scraping project ideas you can share?
#12do custom electronics, robotics, and embedded software development - I specialize in quickly turning ideas into prototypes. I've built custom automation equipment for chemistry labs, sensors that are in use in household/utility applications, control circuitry for construction equipment, 3d printing electronics, data acquisition equipment. No project too small. Few projects too large. Deep discounts for open source hardware work. You can visit this https://goo.gl/29VL2t
Re: Ask HN: Any data scraping project ideas you can share?
#13Here's another idea I just thought of randomly: Scrape and monitor a company's competitor's job listings for them. Some of that data might be difficult to get given the nature of job sites and craigslist and such, but could be interesting to accumulate all of that (including from the company's own site) so you can get an idea when they are hiring. Maybe.
Sorry if I've missed the main idea behind this, but why would that be relevant?
Looking at companies' hiring data is a great way to monitor competition.
If they try to hire for more positions than in the past they're probably growing, conversely they might be stagnating if it's the other way round.
If they hire people with specific skills it might also tell you what they're up to right now like going public or working on a new, supposedly secret project. Take Apple for instance. A notoriously secretive company, previous new projects like the iPhone, the Apple Watch and most notably a self-driving car have first been revealed by their own job postings.
Re: Ask HN: Any data scraping project ideas you can share?
#14Well the obvious scraping business ideas would be hotel rooms and airplane tickets. You might think this area is already saturated but I think there’s still room for disruption and you can capture niche giants like Expedia / Skyscanner / Agoda / Booking don’t handle well. Or you could do b2b with these companies. Also, what about scraping restaurant menus and offering a food search engine?
There's probably a reason those sites don't handle niches very well: There's no scalable business model for those.
Travel / accommodation is a highly competitive as well as intentionally convoluted market. The participants don't want their customers to easily find the best offer. Sites like Expedia or Booking aren't complex and difficult to use because those companies don't know about UX. It's precisely the opposite. Problem is, the goals of their UX most of the time don't align with the users'.
Re: Ask HN: Any data scraping project ideas you can share?
#15Aggregate data on weather and soil composition to identify areas of land around the world most similar to famous wine producing regions.
Wow! That's an awesome idea.
http://www.agrimetrics.co.uk/ there are a few others using land records for farming, not to say there isn't still room for more
Re: Ask HN: Any data scraping project ideas you can share?
#16Am I the only one that worries about licensing and legal issues when it comes to web scraping? I'd be terrified to build a product around it since one law suit would threaten the core business.
Re: Ask HN: Any data scraping project ideas you can share?
#17Scrape product information from ebay and other auction sites. Have a machine learning model that compares auction price vs. real price (or usual auction price). If the auction price is good... buy the products and flip them. In other words, you're basically building a system to help you find the best possible products to flip.
Re: Ask HN: Any data scraping project ideas you can share?
#18Well the obvious scraping business ideas would be hotel rooms and airplane tickets. You might think this area is already saturated but I think there’s still room for disruption and you can capture niche giants like Expedia / Skyscanner / Agoda / Booking don’t handle well. Or you could do b2b with these companies. Also, what about scraping restaurant menus and offering a food search engine?
Can you share any examples/ideas for niches that those sites don't already handle well?
Re: Ask HN: Any data scraping project ideas you can share?
#19Am I the only one that worries about licensing and legal issues when it comes to web scraping? I'd be terrified to build a product around it since one law suit would threaten the core business.
Everything that you can open in the browser you can scrape without any problem. Though keep in mind the number of requests you send to those parties should be thought about as it shouldn't vary greatly compared to a regular user. A user doesn't open 1000 pages in 60 seconds.
Re: Ask HN: Any data scraping project ideas you can share?
#20I have been trying to scrape travel places and recommendations data from TripAdvisor and other travel sites. Read about scrapy in another hn post yesterday and have been trying to get it running. Would appreciate any help on this. sam.xenai [at] gmail [dot] com
Did you check out GitHub for open-source solutions? I'm sure the biggest names in the market are already covered when it comes to scrapers.