Congrats! Web scraping is legal! (US precedent) • Hacker News Board

Re: Congrats! Web scraping is legal! (US precedent)

#71

Post by iso1631 » Wed, Jan 29, 2020, 2:49 PM UTC

TrickyRick wrote: post #34

Pigo wrote: post #16

I always thought scraping was a fun idea, I just couldn't find the right use-case for it. I'm not a sports guy, and the big sites have pretty extensive API's. Something music related would be of interest to me, but I can already get updates on events like concerts (since the monopolies make a fortune selling tickets). I'm not sure what could be useful.

I build https://awardfares.com together with a friend which scrapes airlines' award seat availability. Airlines' websites are horrible from a UX perspective so scraping the data and presenting it in a better way was a pretty obvious use case.

No British Airways?

A quick look shows a lot of award availability to China for some reason...

Re: Congrats! Web scraping is legal! (US precedent)

#72

Post by contravariant » Wed, Jan 29, 2020, 2:49 PM UTC

kabacha wrote: post #22

JaceLightning wrote: post #10

Earlier quoted context omitted.

Probably not. Facts aren't copyrightable but creative works are. So prices on Amazon.com are facts. User reviews are creative so probably copyrighted. Similarly the videos on YouTube are copyrighted. However the number of views and the number of likes are probably scrapable.

See that's where I have problem with this. Isn't data just _data_? Lets draw some pararells to real life. If I go to public space like town square - can't I take pictures, notes and records then go home and draw my analytics from it? What if I read something in a book I bought, can't I quote it? Same thing should be with web resources even if they are creative - as long as I don't publish them I should be able to scr…

This is why I strongly prefer the Dutch term 'auteursrecht' (author's rights) as opposed to copyright. Copyright has this annoying incorrect connotation that it has anything to do with copying when it's really publishing that it should be limiting.

Downloading publicly available data should (by definition of public) not be a violation of someone's rights. However it's easy to see why it wouldn't be desirable for someone to republish creative works as their own, so it's reasonable to give the author control over how their work should be published.

And in the case of price data or similar you would be hard pressed to deem anyone the 'author' of it, hence it would be weird to enforce the author's rights.

Re: Congrats! Web scraping is legal! (US precedent)

#73

Post by levosmetalo » Wed, Jan 29, 2020, 2:49 PM UTC

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

Sure. My data is still my data, and if I publish it on my platform for free, that still shouldn't automatically give you the right to copy the data and provide on your platform.

It's basically the same as a TV broadcasting a film for free, and then going after you legally if you recorded that film and uploaded it to your website.

Re: Congrats! Web scraping is legal! (US precedent)

#74

Post by soup10 » Wed, Jan 29, 2020, 2:50 PM UTC

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

Billions are being made collecting data and content for free and publishing it along with targeted advertisements. Web-scrapers can collect that aggregated data and redistribute it or create competing services. This is terrifying to certain tech giants and threatens their moat's and lock-in/network advantage. They will lobby hard against it with PR campaigns citing everything from security, privacy, copyrights and all sorts of other exaggerated bogeymen. At the end of the day its mostly about preserving their monopolies though.

Re: Congrats! Web scraping is legal! (US precedent)

#75

Post by fouc » Wed, Jan 29, 2020, 2:50 PM UTC

Anyone ever read any stories about the concept of "agents" where everyone had their own computer agents that did all the work and talked to other agents do all the steps like book your flights, tickets, order your food, collect/collate data for your purposes, etc?

We need to make the internet "Agent" friendly.. we should stop assuming the end user (end human?) will ever see any webpage on the internet.

Re: Congrats! Web scraping is legal! (US precedent)

#76

Post by kerkeslager » Wed, Jan 29, 2020, 2:50 PM UTC

xnyan wrote: post #50

TrackerFF wrote: post #45

Earlier quoted context omitted.

I can understand why some do not want scrapers - increased traffic (with practically zero benefits to the owners) is one obvious reason. (Some people will then say "But why not just offer APIs", but that's a lot of extra work and maintenance). It's like with instagram and other social media platforms. The content creators put in the hard work, while the leeches are stealing content for their own benefit, giving zero…

There’s a very effective way out, don’t your data on the public web.

[decided to delete because I misunderstood the context]

Re: Congrats! Web scraping is legal! (US precedent)

#77

Post by candiodari » Wed, Jan 29, 2020, 2:51 PM UTC

powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

https://en.wikipedia.org/wiki/Tortious_interference

This would mostly mean that you cannot start interfering with webscraping you previously allowed merely because you learned that they're making money with the scraped data.

Re: Congrats! Web scraping is legal! (US precedent)

#78

Post by rpedela » Wed, Jan 29, 2020, 2:53 PM UTC

pc86 wrote: post #35

kabacha wrote: post #22

Earlier quoted context omitted.

See that's where I have problem with this. Isn't data just _data_? Lets draw some pararells to real life. If I go to public space like town square - can't I take pictures, notes and records then go home and draw my analytics from it? What if I read something in a book I bought, can't I quote it? Same thing should be with web resources even if they are creative - as long as I don't publish them I should be able to scr…

> Isn't data just data ? No. At the risk of just repeating the comment you didn't understand, creative works are not "just data" - they are copyrightable works that the owner has control over who can use them, not just for profit, but for any reason with few exceptions. You don't just get to drop someone else's work product into your algorithm without their permission.

There are cases where "dropping into your algorithm" would count as fair use such as a search engine of copyrighted content.

Re: Congrats! Web scraping is legal! (US precedent)

#79

Post by AgloeDreams » Wed, Jan 29, 2020, 2:53 PM UTC

kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it.

By People you mean business or people people? Because I don't think people want everything to be public, many in fact use various networks to avoid oversharing and even then many people don't want their old bosses or exes looking at their profiles, there just don't exist tools to limit access that granularly.

Re: Congrats! Web scraping is legal! (US precedent)

#80

Post by scotty79 » Wed, Jan 29, 2020, 2:53 PM UTC

powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

> Seems to imply that every business is somehow beholden to every contract signed by anyone.

Implied contract is that if you publish something, it's public and you have no right to dictate what software people use to consume it.