Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

github.com

Post reply on HN
Print view View original

1–10 of 84 posts

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#2

Post by stingraycharles »

What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#3

Post by notafraudster »

stingraycharles wrote: post #2

What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.

I mostly scrape public interest government data, not commercial data. Personally, I don't really care about laws. I'm not hitting PACER or JSTOR, I'm not starting a competing company, and I'm not making my scrapers available as a service, so I'm totally unlikely to be sued.

In terms of ethics I typically apply a sort of "try to be considerate" test. If I am doing personal, non-commercial scraping, only scraping public data, not releasing the data, using timer delays, respecting 429 codes, and not doing too much to mask my identity, and the servers I'm hitting are massive services handling multi-million user concurrency and I'm using a DigitalOcean droplet, then I'm not really a problem. And I trust their well paid sysadmins to block me or contact me if I am the problem. I do check robots.txt in advance. If there is one, it doesn't stop me, but it generally means I take extra precautions to avoid causing trouble for the service in question.

On the other hand, I once scraped the DPRK's English language press office as part of some research, and about two weeks later it became inaccessible from any country other than Japan, so I'm pretty sure I almost caused a diplomatic incident. Oops.

In this case, it's 40 lines of python code using a single threaded requests request, only scraping a single page, not doing any funny business with user agent spoofing, etc. I think you're right, it's probably not something Instagram wants to happen and theoretically I'm sure they could send a takedown, but I guess I have a pretty laissez-faire attitude about this: is it really a good use of their time to stop this guy?

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#5

Post by dmitriid »

stingraycharles wrote: post #2

What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.

IMO scraping like this is shady at best.

That said, however, there's no straightforward way to work with Instagram. It's original APIs are both limited and locked down. The new Facebookified APIs are limited and next to impossible to work with (they are geared exclusively to ads/marketing). So ¯\_(ツ)_/¯

In a side project I use a library that effectively reverse-engineers Instagram's private API, pretends it's a user using a browser etc.

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#7

Post by donjoe »

fapjacks wrote: post #6

Well, this is just a Python web scraper, and Instagram does in fact attempt to detect and prevent/rate-limit this kind of scraping. They rely very heavily on the source IP to help them determine when to cut you off.

... indeed. Isn't the script 'under the hood' calling the API anyways (looking at 'scrolldown' here)?

I reversed engineered the API myself a couple of weeks ago which was great fun - especially figuring out Instagram's rate limits on interactions such as comments and likes per day/hr.

Post reply on HN