Live data from Hacker News

Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

github.com

41–50 of 84 posts

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#41

Earlier quoted context omitted.

Unfortunately "whether you like it or not" doesn't carry much legal weight.

That's why it is important to root out corruption from law making, nowadays called "lobbying" - this should be illegal. If you expose an endpoint to the public you can't restrict who or what can consume it. You can do throttling on your side but that's it. Otherwise this is just racism towards machines.

Not to mention, this still leaves a hole open as a scraping company can outsource the “scraping” to low-wage data-entry slaves in third-world countries.

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#42
post #35
post #15

Earlier quoted context omitted.

If the data is made available to me as a human, then I am free to delegate the job of retrieving it to a machine if I choose, and I will be doing it whether you like it or not.

I second this opinion. The other point I would make - and of course this doesn't necessarily condone the use of scraping sites that don't want to be scraped - but I suspect scraping is being done on a massive scale - especially with sites such as IG. The influencer marketing model has become such a big one that theres a lot of valuable data to be had that you cant access via the API.

>massive scale

just take every search engine.

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#43
post #15

Earlier quoted context omitted.

If the data is made available to me as a human, then I am free to delegate the job of retrieving it to a machine if I choose, and I will be doing it whether you like it or not.

Unfortunately "whether you like it or not" doesn't carry much legal weight.

This brings up the delightful distinction between malum prohibitum and malum in se.

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#44

What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.

The opinion of one HN’er is that it is definitively a gray area. So whether or not you’re sued for it depends on if someone resourceful has a reason to sue you (e.g. LinkedIn vs HiQ); in most other cases, I don’t think anybody really cares.

There was an earlier case (I don’t recall the details) about scraping violating copyright, which might be a valid defense - that you’re engaged in unauthorized “copying” the (protected) source code (of the webpage), albeit momentarily, into your computer’s memory while your script processes it to extract the relevant contents. So even though it might be fine to copy the image/data/whatever else, you have no right to copy source code - which is certainly protected by copyright (your browser processing the same page is something the copyright owner explicitly allows, by virtue of making it available on the web). By that same token, it can also be considered illegal to save webpages on your harddisk. But do refer to the last sentence in my earlier paragraph.

The LinkedIn vs HiQ case is currently being appealed in the (court of appeals for the) 9th Circuit. Regardless of which way it rules, the decision of the 9th circuit only applies in its the states under its jurisdiction (maybe as a precedent in others, but it won’t be binding). It can still go to Supreme Court, depending on how determined the parties are.

There was a ruling in a different (but somewhat related case) in Jan 2018 that violating the ToS is not a crime https://www.eff.org/deeplinks/2018/01/ninth-circuit-doubles-... To quote: “[T]aking data using a method prohibited by the applicable terms of use”— i.e., scraping — “when the taking itself generally is permitted, does not violate” the state computer crime laws”.

There was a very long discussion on HN about scraping over a year back: https://news.ycombinator.com/item?id=13884357

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#45

What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.

The robots.txt discussion is a worthy one. For example, is it against net neutrality (the concept) to only accept being indexed by Google? I think so. In a world where every site only accepts Google and other well known search engines to index them tgere is no room for a new search engine to appear.

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#46
post #5

Earlier quoted context omitted.

IMO scraping like this is shady at best. That said, however, there's no straightforward way to work with Instagram. It's original APIs are both limited and locked down. The new Facebookified APIs are limited and next to impossible to work with (they are geared exclusively to ads/marketing). So ¯\_(ツ)_/¯ In a side project I use a library that effectively reverse-engineers Instagram's private API, pretends it's a user…

can you provide a link or source for that ? TY

I’m using this one: https://github.com/ping/instagram_private_api

If you search for Instagram Private API, you’ll find implementations for almost any programming language (with aforementioned PHP being the first, probably)

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#47

Earlier quoted context omitted.

Unfortunately "whether you like it or not" doesn't carry much legal weight.

That's why it is important to root out corruption from law making, nowadays called "lobbying" - this should be illegal. If you expose an endpoint to the public you can't restrict who or what can consume it. You can do throttling on your side but that's it. Otherwise this is just racism towards machines.

How do you outlaw lobbying?

Re: Show HN: Instagram-scraper – Scrape instagram photos by tags, without API

#50

Earlier quoted context omitted.

Exactly. Want to block non-humans from automatically accessing your content? OK, then i will block non-humans from deciding what ads to show to me. Want to show me an ad? Have an actual human being manually pick the ad in real time and deliver it to me!

Am I the only one who thinks human non-human divide is conceptually the same as coloured non-coloured divide in the past? I think those who exercise such questions to allow or block access to their services are morally corrupt.

I don’t think we’re There Yet. Once we build machines that can seek out and acquire the resources necessary for their own growth/healing/reproduction, whose rights are worth fighting for, it’ll be easier to make that case.

Anyways, that wouldn’t help here. No machines’ rights are being infringed by preventing you from scraping the web. If said self-reproducing machines were denied access to websites on the grounds that it would be considered “scraping”, I’d be at the front of the march to legislate against that kind of discriminatory behavior; but that’s not what we have here.

I do think that, given the obvious potential for both value and harm in scraping, it might make sense to provide licenses for demonstrably non-harmful scrapers.

Post reply on HN