RIP Aaron Swartz.
Congrats! Web scraping is legal! (US precedent)
121–130 of 409 posts
Re: Congrats! Web scraping is legal! (US precedent)
#122"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…
Linkedin want their data to be scraped by bots so they have to keep it public, otherwise you wouldn't find peoples profile from Google. They just don't want bots from from their competitors like hiQ to scrape it.
Re: Congrats! Web scraping is legal! (US precedent)
#123German copyright has the concept of a "Datenbankwerk" (since the 90s). E.g. the telephone book contains lots of boring facts that are each in themselves not copyrightable. However the collection in itself is copyrightable, as it required substantial effort to create. It seems odd that US copyright law wouldn't have a similar provision, or that it doesn't apply here?
Also known as "sweat of the brow." The US Supreme Court rejected such protections in the 90s. I'm sure it's complicated, but copyrights are not presumed.
https://en.wikipedia.org/wiki/Sweat_of_the_brow#United_State...
Re: Congrats! Web scraping is legal! (US precedent)
#124Earlier quoted context omitted.
Probably not. Facts aren't copyrightable but creative works are. So prices on Amazon.com are facts. User reviews are creative so probably copyrighted. Similarly the videos on YouTube are copyrighted. However the number of views and the number of likes are probably scrapable.
See that's where I have problem with this. Isn't data just _data_? Lets draw some pararells to real life. If I go to public space like town square - can't I take pictures, notes and records then go home and draw my analytics from it? What if I read something in a book I bought, can't I quote it? Same thing should be with web resources even if they are creative - as long as I don't publish them I should be able to scr…
Your examples are really wanting greater freedom to copy rather than about the distinction between data and creative work. Copyright is supposed to encourage people to make creative work, not encourage people to record existing facts. I think this distinction is important because creative work isn't actually necessary to anyone else - they could create their own different one if they wanted. But data might only have one correct value and if that was locked away by copyright, it would limit other people's ability to do things that can't be done with some different data.
Re: Congrats! Web scraping is legal! (US precedent)
#125Since this is probably granted for anyone with a technical understanding, it is nice to see that the legislative powers are on board with this.
Re: Congrats! Web scraping is legal! (US precedent)
#126Earlier quoted context omitted.
I think any ruling that says LinkedIn can't put in protectionary measures against automated requests is doomed to be overturned, as long as they're not doing it discriminately. Captcha, rate limiting, user agent testing, etc are all common tools to protect against malicious/unintentional denials of service. The question is what was LinkedIn doing, and did it specifically target hiQ while permitting others of the same…
Why would it be an issue if it is discriminatory? Linkedin can use its servers any way they like, unless they ve promised their users that their data can be scraped indiscriminately
Because of the court case. This is just an injunction pending an actual decision.
Re: Congrats! Web scraping is legal! (US precedent)
#127The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.
One of my clients is involved in property tax collection and reporting. Property Tax records are public info, and their website allows looking up the records for any property without a login. However, the data behind this website it the _source_ of the public records, and not the public records themselves (which would be local government databases). For years now we've been in an arms race with someone using a botnet…
just put an option to download the raw csvs buried somewhere there. someone who is putting in the effort to bot scrapers will find that link, and save your server the load.
Re: Congrats! Web scraping is legal! (US precedent)
#128The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.
One of my clients is involved in property tax collection and reporting. Property Tax records are public info, and their website allows looking up the records for any property without a login. However, the data behind this website it the _source_ of the public records, and not the public records themselves (which would be local government databases). For years now we've been in an arms race with someone using a botnet…
Would rate limiting be a viable solution?
Re: Congrats! Web scraping is legal! (US precedent)
#129"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…
[deleted]
Re: Congrats! Web scraping is legal! (US precedent)
#130This case does not make web scraping legal or illegal, it just sets a precedent that Computer Frauds and Abuse Act (CFAA 1986) cannot directly be applied in web scraping of public data for fair use. Web scraping can be legal with caveats which is, as long as you are just scraping a public information and do not re-sale it for profit without significant transformation and value addition i.e. a fair use doctrine is applied.
If you use bots to login using a username and password and than scrap the information, it's still wrong and infringement, as action of login binds you to an implicit contract with this website and by logging in you accept those terms. The ruling on HiQ vs LinkedIn is quite nuanced and if you are crawling and than repackaging that information and selling it somehow then there is a highly likely chance the precedent in this case won't apply.
Technical barriers like rate limiting and captcha are legitimate way to guard against not just web crawlers but denial of service (DoS) attacks, so it's about how it's been put forward by lawyer. So in general website can still continue to block like google, facebook, amazon and all the big sites are doing it. It's not illegal, I found a better deep understanding of this case based on explanation from a EFF. [1] [2]
[1] https://www.eff.org/deeplinks/2019/09/victory-ruling-hiq-v-l...
[2] https://www.eff.org/document/hiq-v-linkedin-ninth-circuit-de...