11–20 of 84 posts
Does not really work... downloads the same 5 pictures over and over again.
Thanks!
Well, this is just a Python web scraper, and Instagram does in fact attempt to detect and prevent/rate-limit this kind of scraping. They rely very heavily on the source IP to help them determine when to cut you off.
Well, this is just a Python web scraper, and Instagram does in fact attempt to detect and prevent/rate-limit this kind of scraping. They rely very heavily on the source IP to help them determine when to cut you off.
... indeed. Isn't the script 'under the hood' calling the API anyways (looking at 'scrolldown' here)? I reversed engineered the API myself a couple of weeks ago which was great fun - especially figuring out Instagram's rate limits on interactions such as comments and likes per day/hr.
What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.
What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.
What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.
I mostly scrape public interest government data, not commercial data. Personally, I don't really care about laws. I'm not hitting PACER or JSTOR, I'm not starting a competing company, and I'm not making my scrapers available as a service, so I'm totally unlikely to be sued. In terms of ethics I typically apply a sort of "try to be considerate" test. If I am doing personal, non-commercial scraping, only scraping publi…
I understand what you're getting at, but if someone tells you explicitly not to do something (e.g. in the terms of service of their website), doing that thing anyway doesn't seem very considerate.
What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.
If the data is made available to me as a human, then I am free to delegate the job of retrieving it to a machine if I choose, and I will be doing it whether you like it or not.
Earlier quoted context omitted.
If the data is made available to me as a human, then I am free to delegate the job of retrieving it to a machine if I choose, and I will be doing it whether you like it or not.
Unfortunately "whether you like it or not" doesn't carry much legal weight.
Not saying this is right, but if the service provider wants to play cat and mouse then I’m happy to take part.
What is HN’s opinion on the legality of these types of scrapers? Instagram’s robots.txt disallows this kind of scraping, same for their ToS. Legal precedents have been mixed - the recent LinkedIn vs HiQ case is a good signal, but it’s still in appeals court.