Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Any data scraping project ideas you can share?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Ask HN: Any data scraping project ideas you can share?

#21

Post by peternicky »

BjoernKW wrote: post #13
gyvastis wrote: post #8

Earlier quoted context omitted.

Sorry if I've missed the main idea behind this, but why would that be relevant?

Looking at companies' hiring data is a great way to monitor competition. If they try to hire for more positions than in the past they're probably growing, conversely they might be stagnating if it's the other way round. If they hire people with specific skills it might also tell you what they're up to right now like going public or working on a new, supposedly secret project. Take Apple for instance. A notoriously se…

This is in my opinion the same as buying at the height of a bubble; by the time you get this data on your competitor, you will be way behind. Why not spend resources on improving the offering?

Re: Ask HN: Any data scraping project ideas you can share?

#22

Post by anywherenotes »

dchuk wrote: post #18
richardknop wrote: post #7

Well the obvious scraping business ideas would be hotel rooms and airplane tickets. You might think this area is already saturated but I think there’s still room for disruption and you can capture niche giants like Expedia / Skyscanner / Agoda / Booking don’t handle well. Or you could do b2b with these companies. Also, what about scraping restaurant menus and offering a food search engine?

Can you share any examples/ideas for niches that those sites don't already handle well?

I just booked a vacation on Expedia. What I really wanted was to find a list of rooms by price (airfare included) which can comfortably sleep 4 people - so I needed 4 actual beds. I tried looking at bigger rooms, but it looked like booking 2 cheap rooms was less expensive, but I'm still not sure if that's true. So basically you could see if you can accommodate large parties. Also, when I booked the rooms, I think it made me select one type, but in theory, i might have wanted one ocean front room and one without the view.

Re: Ask HN: Any data scraping project ideas you can share?

#23

Post by anywherenotes »

gyvastis wrote: post #19
remyp wrote: post #16

Am I the only one that worries about licensing and legal issues when it comes to web scraping? I'd be terrified to build a product around it since one law suit would threaten the core business.

Everything that you can open in the browser you can scrape without any problem. Though keep in mind the number of requests you send to those parties should be thought about as it shouldn't vary greatly compared to a regular user. A user doesn't open 1000 pages in 60 seconds.

don't most sites claim they own data? Like could you legally scrape reddit and make your own site?

Re: Ask HN: Any data scraping project ideas you can share?

#24

Post by ambivalents »

gyvastis wrote: post #8
dchuk wrote: post #4

Here's another idea I just thought of randomly: Scrape and monitor a company's competitor's job listings for them. Some of that data might be difficult to get given the nature of job sites and craigslist and such, but could be interesting to accumulate all of that (including from the company's own site) so you can get an idea when they are hiring. Maybe.

Sorry if I've missed the main idea behind this, but why would that be relevant?

If you're unhappy with your current workplace and want to do similar work in the same industry, competitors are a great place to start.

Re: Ask HN: Any data scraping project ideas you can share?

#25

Post by BjoernKW »

peternicky wrote: post #21
BjoernKW wrote: post #13

Earlier quoted context omitted.

Looking at companies' hiring data is a great way to monitor competition. If they try to hire for more positions than in the past they're probably growing, conversely they might be stagnating if it's the other way round. If they hire people with specific skills it might also tell you what they're up to right now like going public or working on a new, supposedly secret project. Take Apple for instance. A notoriously se…

This is in my opinion the same as buying at the height of a bubble; by the time you get this data on your competitor, you will be way behind. Why not spend resources on improving the offering?

True. Still, keeping tabs on the competition is big business.

Re: Ask HN: Any data scraping project ideas you can share?

#27

Post by jonesgrant »

PLEASE READ!!!! Hello Guys,This is a Life Time transformation !!!Am so happy I got mine from Kelvin Roland. My blank ATM card can withdraw $3,300 daily. I got it from him last week and now I have $16,500 for free for just a week. The blank ATM CARD withdraws money from any ATM machines and there is no name on it, it is not traceable and now i have money for business and enough money for me and my family to live on. I am really happy i met Kelvin Roland because i met two people before him and they took my money not knowing that they were scams. But am happy now. Kelvin Roland sent the card through DHL and i got it in two days. Get your own card from him now he is not like other scammer pretending to have the ATM card, he is giving it out for free to help people even if it is illegal but it helps a lot and no one ever gets caught. i'm grateful to Kelvin Roland because he changed my story all of a sudden. The card works in all countries except Philippines and Mali. Kelvin Roland email address is kelvinroland.blankatm@outlook.com!

Re: Ask HN: Any data scraping project ideas you can share?

#28

Post by zapperdapper »

remyp wrote: post #16

Am I the only one that worries about licensing and legal issues when it comes to web scraping? I'd be terrified to build a product around it since one law suit would threaten the core business.

Me too. Just look at the legal battle between LinkedIn and hiQ.

I have scraped, but not on the level where it would draw attention. Use the APIs if you can - Trip Advisor do offer an API, as do Bing (for search results).

Unless you are scraping at huge scale I actually think the bigger problem is lack of semantics - people like diffbot[1] using AI/ML to try and solve this issue.

[1] - https://www.diffbot.com

Re: Ask HN: Any data scraping project ideas you can share?

#29

Post by zapperdapper »

Some of the scraping projects I've done in the past have been where the article content I wanted was on a large site with great content but the site was awful to read - due to horrible combinations of pop-ups, colour schemes, adverts etc etc. I would spider and download content, process, and build my own database/simple CMS to make reading the content offline a much better experience.

Are you just looking for a personal project something like that might work for you...

Re: Ask HN: Any data scraping project ideas you can share?

#30

Post by dhruvkar »

Searching for flights using rewards miles.

I think United & American used to have APIs that were shut down, so you'd need to scrape account data and flights. It would work best as a desktop app. Other airlines have APIs, but not sure how deep they are.

Huge pain point, especially when trying to combine different rewards programs.

Post reply on HN