Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

241–250 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#241

Earlier quoted context omitted.

> Lots of sites have ToS preventing such things, are those legally void now? Are captchas on public pages illegal, even if you request the page 8000 times in a second? ToS are subservient to the law; you can (probably) terminate a service account from a user that breaks your ToS, but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS do…

> but if the user does not have a service account (as is the case for HiQ, it doesn't seem they were using accounts for it), then your ToS does not apply, since you've technically not entered a binding legal contract with them. Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account. I have interpreted the LinkedIn…

Terms of Service is a form of contractual agreement, which requires there be an offer and subsequent agreement by the parties.

I don't think criminal law was ever part of this.

Re: Congrats! Web scraping is legal! (US precedent)

#242
post #237
post #213

Earlier quoted context omitted.

The issue here for some, if not many, is a matter of scale. It is one thing if an end-user, whom I am trying to service, comes to my site and gets my publicly available data. Maybe I monetize with ads, maybe not. It doesn't matter, that is the audience I am trying to service, regardless of size. But when you scrape it my load goes up dramatically. A load I have to pay for. It is analogous to the privacy debates going…

Provide an API for public data to reduce the costs associated with rendering a full blown page, and deliver just the information needed.

Entirely feasible. Also reasonable for you to pay me for the service as it is taking my development efforts to meet your business model. The advantage to you is you have a defined interface that I won't prevent.

Re: Congrats! Web scraping is legal! (US precedent)

#243
post #218

Earlier quoted context omitted.

> Would you be OK with a company scraping your blog and selling it? Selling it how? If they put my blog posts in a book and try to sell that book, that’s copyright infringement. If they put my blog posts in an ML model corpus to train a translation service, and they then charge pay-per-use access to the resulting service... I don’t think I’d care, nor do I think there’s anything morally or legally wrong with that. If…

Why can't I have terms on my website that say how you can use my information? Examples where this is allowed: - Images/media (Creative commons) - Code (Open source licenses) You say it isn't allowed for: - Personal data Unless I'm misunderstanding your philosophy (which seems to say copyright is OK, but public information must be public to all): You believe that it's morally OK for me to prevent a company selling my…

> Why can't I have terms on my website that say how you can use my information?

Neither Creative Commons nor copyleft (nor copyright in general!) can assert anything about private use. IP rights are commercial rights; they affect sellers of your IP. They don’t affect end-consumers of your IP.

Note that even the GPL can’t force someone to publish the source of their GPLed-library-containing program, if they never publish the program itself, but only build it for their own private use.

Why? Because, by broadcasting the code of your GPLed library, you granted people an implicit use-right to it! Not a redistribution right; not a derivative-works right; but a use right. (If this wasn’t true, then people would be breaking the law by reading “common” newspapers in a cafe, or by listening to the radio, since they never entered into any explicit contract with the distributor/broadcaster.)

How does software licensing work, then? Mostly by 1. companies installing software on computers for their employees to use being considered IP redistributors; and 2. attachment of copyright through sampling when asset samples [e.g. brushes/textures in Photoshop] are distributed through the program. Other than that, there’s really no law forcing end-users to pay for software licenses. This is why e.g. WinRAR would never have been able to sue anybody. They published their shareware binary (without gating it behind a contractual relationship, like Adobe’s Creative Cloud installer); so now you have a use-right to it!

> You believe that it's morally OK for me to prevent a company selling my book, but not morally OK for me to prevent a company selling my phone number?

Copyright exists because your ability to make money from your own creative works hinges on your ability to exclusively license those works. If a publisher can get a redistribution license to your manuscript for free from a third party, why would they buy it from you?

You having exclusive access to your phone number does not make you money; others having access to your phone number does not deprive you of money you could have made by keeping that information private. Thus, there’s no advantage to introducing IP law into this domain (the domain of facts.)

There was a recent court case about someone creating a subway map by copying the raw data from existing subway maps, where the comments went deeper into this.

Re: Congrats! Web scraping is legal! (US precedent)

#244

Earlier quoted context omitted.

> Are you sure about this? I am not a lawyer, but I believe that the Terms of Service applies to all users, not just those that explicitly set up a user account. How would that even work? If I browse to any random public page of your website, it's served to me before you've even transmitted the terms of service. How could I be bound by those terms of service when I haven't even seen them?

IANAL, but it seems like ToS could still govern your use of the data which you viewed. Sure, it seems like you couldn't claim any violation based on visiting a random page. But if the ToS is clearly identified on the page and you do something with the data that violates them, perhaps the owner of the site has a case.

> perhaps the owner of the site has a case.

Except it sounds like the owner doesn't. If the information is on the page made public, the owner of the page can't place terms on what is done with the data downstream. They'd have to implement some real binding system such as authentication where CFAA would apply. (IANAL)

Re: Congrats! Web scraping is legal! (US precedent)

#245
post #40

Earlier quoted context omitted.

There's plenty of grey here. For example, scrapers that try to check people in for flights to get better seats. Some that tried to charge for that. That creates problems, where some customers benefit at the expense of others, high load on a "locking type" piece of code, etc. Similar for ticket sales for concerts, and probably other spaces. There are also companies that provide added value by compiling and correlating…

I would think, and of course could be wrong, it would be as legal as Google scraping all of the web sites that they do in order to create their search engine in the first place. In particular, Google provides cached versions of web pages. That's pretty hardcore scraping.

[deleted]

Re: Congrats! Web scraping is legal! (US precedent)

#246
post #203

Earlier quoted context omitted.

> but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free. It's not free, it takes time to collect data. Buying it makes a lot of sense as long as you pay less than what's your own time worth to you...

It is true in the current situation, though I would prefer that we ensure free data must be free. In that case buyers of data would be incentivized to pressure providers of free data to improve the data quality.

The data does remain free, as long as LinkedIn still provides it for free.

The data without the noise is what you're paying for. The service of winnowing out what you care about from what you don't care about.

Considering how big of an effort it is, and that the source from which it came is still available, why should the cleaned data be free? If I collect fallen trees from public land, chop it into usable firewood, should my bundles of firewood also be free? Or I collect solar power with my own solar cells, should I have to give you the electricity for free?

Re: Congrats! Web scraping is legal! (US precedent)

#247
post #40

Earlier quoted context omitted.

There's plenty of grey here. For example, scrapers that try to check people in for flights to get better seats. Some that tried to charge for that. That creates problems, where some customers benefit at the expense of others, high load on a "locking type" piece of code, etc. Similar for ticket sales for concerts, and probably other spaces. There are also companies that provide added value by compiling and correlating…

I would think, and of course could be wrong, it would be as legal as Google scraping all of the web sites that they do in order to create their search engine in the first place. In particular, Google provides cached versions of web pages. That's pretty hardcore scraping.

I put "legal" in quotes because I assume it's not criminal. It would, though, likely result in some pretty quick civil action.

Re: Congrats! Web scraping is legal! (US precedent)

#248
post #203

Earlier quoted context omitted.

> but enough people think of it as a valuable enough service to pay for; even if they know they could get it themselves, for free. It's not free, it takes time to collect data. Buying it makes a lot of sense as long as you pay less than what's your own time worth to you...

It is true in the current situation, though I would prefer that we ensure free data must be free. In that case buyers of data would be incentivized to pressure providers of free data to improve the data quality.

What do you mean free data must be free?

The data is free, but the aggregated formatted data has been worked on and processed, are you saying the resulting aggregated data should also be free? That isn't going to happen, why would anyone do that work for free?

Or are you afraid linkedIn and others will make everything private? That's completely up to linkedIn or individual linkedIn users what they want to make private vs public. Maybe more data would be made private if they don't want it scraped. I don't think that's inherently a good or bad thing.

Re: Congrats! Web scraping is legal! (US precedent)

#249

So many ideas start to come to mind if scraping is legal. Can we start to scrape Google Search in order to bootstrap building an alternative to Google Search? Search is a really hard problem (that somebody should tackle), but if we can leverage what Google has already scraped from the web and associated with popular search terms, we can use that to help train and validate our search model. Can we scrape Reddit, Twitt…

Seems like a hard problem to legally solve. I can see so many valid use cases for bots to scrape pages. But in all of your examples, I'm inclined to say that it shouldn't be allowed.

Maybe it falls into a "fair use" situation? Obviously copying an entire website would not be considered fair use, but something like scraping a bunch of public profiles on Steam to get aggregate data on what games are played the most seems totally valid.

Hopefully it doesn't end up with everything gated behind a sign-in and a TOS.

Re: Congrats! Web scraping is legal! (US precedent)

#250
post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

Sure. My data is still my data, and if I publish it on my platform for free, that still shouldn't automatically give you the right to copy the data and provide on your platform. It's basically the same as a TV broadcasting a film for free, and then going after you legally if you recorded that film and uploaded it to your website.

This does not legalize theft, it says sites cannot respond to suspected scrapers differently than they respond to non-suspected scrapers. You can still rate limit, as long as you do it universally to all site users

Copyright law is unchanged. If someone scrapes your blog and then re-uses your posts on their own blog, you still have possible copyright infringement claim

Post reply on HN