Congrats! Web scraping is legal! (US precedent) • Hacker News Board

Re: Congrats! Web scraping is legal! (US precedent)

#31

Post by Nitramp » Wed, Jan 29, 2020, 2:33 PM UTC

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

"public" is not the right concept here I think. E.g. imagine a composer conducting a public airing of some work of music (e.g. on some festival). That you were able to hear the music in public doesn't mean the {composer,artists,...} give up their copyrights.

Re: Congrats! Web scraping is legal! (US precedent)

#32

Post by TrickyRick » Wed, Jan 29, 2020, 2:33 PM UTC

thomasjames wrote: post #23

This wouldn't supersede any violations of the ToS, though. I guess it would mean they can't sue you (maybe?) but they could still kick you off their service. Could they still go after you for breach of contract?

Wasn't the point that they were scraping public profiles, i.e. they never signed up and thereby never signed any ToS?

Re: Congrats! Web scraping is legal! (US precedent)

#33

Post by mantap » Wed, Jan 29, 2020, 2:34 PM UTC

chii wrote: post #12

JaceLightning wrote: post #10

Earlier quoted context omitted.

Probably not. Facts aren't copyrightable but creative works are. So prices on Amazon.com are facts. User reviews are creative so probably copyrighted. Similarly the videos on YouTube are copyrighted. However the number of views and the number of likes are probably scrapable.

> User reviews are creative so probably copyrighted. I wonder if the number of stars are copyrighted. It's not creative, but a fact.

Probably not since each star review is a separate "work" by a separate author. Mechanically combining multiple non-copyrightable things into one doesn't make it copyrightable. If Amazon arranged their users' star reviews into an infographic that would be copyrightable.

Re: Congrats! Web scraping is legal! (US precedent)

#34

Post by TrickyRick » Wed, Jan 29, 2020, 2:35 PM UTC

Pigo wrote: post #16

I always thought scraping was a fun idea, I just couldn't find the right use-case for it. I'm not a sports guy, and the big sites have pretty extensive API's. Something music related would be of interest to me, but I can already get updates on events like concerts (since the monopolies make a fortune selling tickets). I'm not sure what could be useful.

I build https://awardfares.com together with a friend which scrapes airlines' award seat availability. Airlines' websites are horrible from a UX perspective so scraping the data and presenting it in a better way was a pretty obvious use case.

Re: Congrats! Web scraping is legal! (US precedent)

#35

Post by pc86 » Wed, Jan 29, 2020, 2:35 PM UTC

kabacha wrote: post #22

JaceLightning wrote: post #10

Earlier quoted context omitted.

Probably not. Facts aren't copyrightable but creative works are. So prices on Amazon.com are facts. User reviews are creative so probably copyrighted. Similarly the videos on YouTube are copyrighted. However the number of views and the number of likes are probably scrapable.

See that's where I have problem with this. Isn't data just _data_? Lets draw some pararells to real life. If I go to public space like town square - can't I take pictures, notes and records then go home and draw my analytics from it? What if I read something in a book I bought, can't I quote it? Same thing should be with web resources even if they are creative - as long as I don't publish them I should be able to scr…

> Isn't data just data?

No. At the risk of just repeating the comment you didn't understand, creative works are not "just data" - they are copyrightable works that the owner has control over who can use them, not just for profit, but for any reason with few exceptions.

You don't just get to drop someone else's work product into your algorithm without their permission.

Re: Congrats! Web scraping is legal! (US precedent)

#36

Post by Nitramp » Wed, Jan 29, 2020, 2:35 PM UTC

German copyright has the concept of a "Datenbankwerk" (since the 90s).

E.g. the telephone book contains lots of boring facts that are each in themselves not copyrightable. However the collection in itself is copyrightable, as it required substantial effort to create.

It seems odd that US copyright law wouldn't have a similar provision, or that it doesn't apply here?

Re: Congrats! Web scraping is legal! (US precedent)

#37

Post by alaaf » Wed, Jan 29, 2020, 2:35 PM UTC

Wowfunhappy wrote: post #29

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

I want web scraping to be legal—but, is it really contradictory to say "I want this data to be accessible to real humans only"? Any person can post on Hacker News. However, if someone made a bot to post to Hacker News, I think most of us would be pretty upset.

Isn’t this different from read access? Apart from server resources, downloading content doesn’t effect a web site as much as posting on it.

Re: Congrats! Web scraping is legal! (US precedent)

#38

Post by powrtoch » Wed, Jan 29, 2020, 2:36 PM UTC

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site."

Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawling their sites. Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second?

"In this case, hiQ argued that LinkedIn’s technical measures to block web scraping interfere with hiQ’s contracts with its own customers who rely on this data. In legal jargon, this is called” malicious interference with a contract”, which is prohibited by American law."

This is almost weirder. If LinkedIn wanted to force users to sign in to view profile info, would they be not allowed to do that because some company had signed a contract that implicitly assumed access to that data? If someone writes a web scraper for my site, and I unknowingly change my site in a way that breaks that scraper, can a court force me to revert the change?

Seems to imply that every business is somehow beholden to every contract signed by anyone.

Re: Congrats! Web scraping is legal! (US precedent)

#39

Post by mokus » Wed, Jan 29, 2020, 2:36 PM UTC

Nitramp wrote: post #31

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

"public" is not the right concept here I think. E.g. imagine a composer conducting a public airing of some work of music (e.g. on some festival). That you were able to hear the music in public doesn't mean the {composer,artists,...} give up their copyrights.

I think copyright as you mention here is the right concept, or at least a lot closer. In particular, the limits on copyright. If someone is reciting a list of facts in public, they can’t expect people not to record those facts, because copyright doesn’t apply to that. Reciting the list in public using computers shouldn’t change that.

Re: Congrats! Web scraping is legal! (US precedent)

#40

Post by tyingq » Wed, Jan 29, 2020, 2:37 PM UTC

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

There's plenty of grey here. For example, scrapers that try to check people in for flights to get better seats. Some that tried to charge for that. That creates problems, where some customers benefit at the expense of others, high load on a "locking type" piece of code, etc. Similar for ticket sales for concerts, and probably other spaces.

There are also companies that provide added value by compiling and correlating "public info" in a useful way that creates value. If Google let me scrape their search and remove ads, it would be popular, but is it "legal"? Or maybe Google Maps?