Show HN: Instagram-scraper – Scrape instagram photos by tags, without API • Hacker News Board

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#81

Post by Jonovono » Mon, Jun 25, 2018, 2:57 PM UTC

I wish Instagram had an API for their photos. You used to be able to query photos by coordinates and what not. I started building a weather app that would only show you the feels like temperature and wanted to show some photos of what it looked like outside near your location. Instagram kept rejecting my app and then they shut down the API completely. So what I had to do is this:

Query Facebook for a list of location IDs near your location -> use those location IDs to get photos tagged with that location on Instagram -> wait for response for all of those photos to come back and then sort by recently taken. It ends up taking fairly long.

I still made the app anyway: https://itunes.apple.com/ca/app/feels-see-what-it-feels-like... but I am going to transition to using public Snapchat stories.

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#82

Post by Jonovono » Mon, Jun 25, 2018, 2:59 PM UTC

stingraycharles wrote: post #2

What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.

I made an app that used Tinders private API and got a letter from their lawyer. I didn't fight it, but I did a bit of research and it seems like I wouldn't have won haha

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#83

Post by katzgrau » Mon, Jun 25, 2018, 5:45 PM UTC

aleph- wrote: post #76

katzgrau wrote: post #56

If you've ever tried scraping Facebook you'd know that it's nearly impossible to do so reliably. They have a formidable anti-scrape strategy. Instagram is currently ridiculously easy to scrape - but I doubt it's going to be that way for long.

Any links on their anti scrape strategy? Been developing a distributed scraper with baked in proxy per process ability to harvest data for a data project I'm working on. So this would be good to know.

I don't think it's something they would publicize otherwise it would be an invite to work around it. The best route might be to try and scrape a public business page (better yet, several hundred) - that'll show you everything you need to know

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#84

Post by fapjacks » Tue, Jun 26, 2018, 11:18 AM UTC

anonu wrote: post #49

r0f1 wrote: post #10

Does not really work... downloads the same 5 pictures over and over again.

There are dozens and dozens of similar projects on Github and elsewhere...

Right, this is what I was driving at. This does not belong on HN, as it is neither interesting nor novel. And it's self-promotion which, when it's also completely unimpressive, stinks even more in my opinion. If the OP had posted a link to his blog which explained his ML project in detail -- and it was an interesting project and not for example an Intro to ML Coursera project or something equally lame -- then perhaps it might be appropriate for HN. This is not that.