Ask HN: Any data scraping project ideas you can share? • Hacker News Board

Re: Ask HN: Any data scraping project ideas you can share?

#31

Post by ezekg » Sat, Nov 18, 2017, 1:39 AM UTC

dhruvkar wrote: post #30

Searching for flights using rewards miles. I think United & American used to have APIs that were shut down, so you'd need to scrape account data and flights. It would work best as a desktop app. Other airlines have APIs, but not sure how deep they are. Huge pain point, especially when trying to combine different rewards programs.

I had something similar awhile back but it was eventually shut down by the airline’s legal department. If it’s not provided via a public API, I doubt scraping will turn out any different than my project.

Re: Ask HN: Any data scraping project ideas you can share?

#32

Post by dhruvkar » Sat, Nov 18, 2017, 1:55 AM UTC

ezekg wrote: post #31

dhruvkar wrote: post #30

Searching for flights using rewards miles. I think United & American used to have APIs that were shut down, so you'd need to scrape account data and flights. It would work best as a desktop app. Other airlines have APIs, but not sure how deep they are. Huge pain point, especially when trying to combine different rewards programs.

I had something similar awhile back but it was eventually shut down by the airline’s legal department. If it’s not provided via a public API, I doubt scraping will turn out any different than my project.

If it's a desktop app and the crawling is not centralized, I doubt they'd be able to do much about it.

Lot of crawl-heavy SEO tools work this way.

Re: Ask HN: Any data scraping project ideas you can share?

#33

Post by ezekg » Sat, Nov 18, 2017, 2:33 AM UTC

dhruvkar wrote: post #32

ezekg wrote: post #31

Earlier quoted context omitted.

I had something similar awhile back but it was eventually shut down by the airline’s legal department. If it’s not provided via a public API, I doubt scraping will turn out any different than my project.

If it's a desktop app and the crawling is not centralized, I doubt they'd be able to do much about it. Lot of crawl-heavy SEO tools work this way.

Idk, I had a free open source command line project that scraped flight data and that got shut down. It might have been the particular airline, though, because they specifically disallow scraping from all third-parties including eg Google Flights, SkyScanner, etc.

Re: Ask HN: Any data scraping project ideas you can share?

#34

Post by rosha » Sun, Nov 19, 2017, 2:25 PM UTC

I recently had a similar idea which I ended up putting into real world, so I built a small search engine "UkookU" for used vehicles from scratch, it did not take me a long time to get the data and maintain it as I did not need to build any scrapers or care about sites blocking/abuse as I used a third party scrapping API called ProxyCrawl https://proxycrawl.com which allows me to make a proof of concept of the idea for free, so that shortened the time I needed to build the hadoop engine, etc.

I am thinking now recently to build a recruiting talent pool service, which is based on aggregated data from LinkedIn, Google, Bing, Yahoo, Facebook and many other sites and I am pitching something around it as I can get all the data with ProxyCrawl.

I am also thinking recently to do something about keyword ontologies, small markets around me using google/yandex data and offer it as a free helpful tool in the form of a mobile app. How do you normally get your data for your projects?

Re: Ask HN: Any data scraping project ideas you can share?

#35

Post by dhruvkar » Tue, Nov 21, 2017, 4:24 PM UTC

ezekg wrote: post #33

dhruvkar wrote: post #32

Earlier quoted context omitted.

If it's a desktop app and the crawling is not centralized, I doubt they'd be able to do much about it. Lot of crawl-heavy SEO tools work this way.

Idk, I had a free open source command line project that scraped flight data and that got shut down. It might have been the particular airline, though, because they specifically disallow scraping from all third-parties including eg Google Flights, SkyScanner, etc.

Interesting. As a consumer, I'd love to see something in that space. I hate paying for most things, but a desktop app that allows me to search for rewards miles would be something I would pay a yearly fee for.