Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

361–370 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#361

Earlier quoted context omitted.

What if the organization is one person in an LLC? Do they get rights? If so then a big company can hire a bunch of little LLCs to act as rights-having proxies for any task that requires them.

I'm going to assume you're asking in good faith and try to address the confusion here. The human does get rights, the organization doesn't. In some cases, believing that humans have rights and believing that organizations have rights might lead one to the same action. In those cases, I'd take the action. I wouldn't want to violate a human's rights out of some vindictive dislike of organizations: that's not the point.…

> The human does get rights, the organization doesn't.

Organisations are (usually) legal persons, too; they just have fewer responsibilities, get fewer rights in exchange.

Re: Congrats! Web scraping is legal! (US precedent)

#362

Dangerously misleading headline. There is a circuit split on the interpretation of the CFAA, including on what counts as “unauthorized access”. If you are sued outside of the Ninth Circuit, you might get very different results.

A circuit split? Is there another circuit with a contradictory ruling?

Re: Congrats! Web scraping is legal! (US precedent)

#363

Earlier quoted context omitted.

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want i…

> I don't believe organizations have rights, period So a group of people, joining together in a common cause, don’t have rights as members of that group? You are contradicting yourself. Organizations are simply groups of people with a shared cause. To deny rights to the organization, you necessarily have to deny personal rights. Example: John and Sam form an organization for comic book collecting. They have a secret…

> So a group of people, joining together in a common cause, don’t have rights as members of that group?

Emphasis added. No, they don't. They have rights as individual people.

> [...] this violates the right of Sam to freely associate with John and his right to not have to associate with the interloper. It also violates the privacy of San and John as individuals [...]

Yes, that. "Sam" and "John" would have cause for civil (and probably criminal) action against the interloper. "John and Sam, Inc" has no say in the matter.

It might be useful to grant John and Sam, Inc the privilege to own property, but even that isn't actually a right except insofar as Sam and John have a right not have the value of their assets (ie 50% ownership of John and Sam, Inc - functioning as a proxy for ownership of various comics) actively sabotaged/vandalized.

Re: Congrats! Web scraping is legal! (US precedent)

#364

Earlier quoted context omitted.

Terms of Service is a form of contractual agreement, which requires there be an offer and subsequent agreement by the parties. I don't think criminal law was ever part of this.

From the article, the LinkedIn decision was that scraping data does not violate the Computer Fraud and Abuse Act. Violating that act was considered to be criminal activity. ( https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act )

But the claim of a violation was only a claim as part of a civil trial. The law has both civil and criminal elements to it, and this is about the tort part of the law.

LinkedIn made threats accusing hiq of criminal behavior, but that doesn't mean there's any criminal precedent being set here, as far as I can tell. And no one was criminally charged.

Separately, part of the ruling states that for the purposes of authorization, defying a cease and desist letter does not constitute illegal access, which might have some criminal implications. They imply some sort of technical authorization system must be bypassed, which didn't happen, since the data is "public."

(Which doesn't square well, imho, with existing meatspace law. If a public serving business banned someone from their store, the door being unlocked isn't an excuse to ignore that ban and trespass. But I digress.)

With the overlapping areas of law, it's admittedly beyond my understanding. But the law is generally viewed, like dmca, as being overreaching, if not at least partly unconstitutional.

Re: Congrats! Web scraping is legal! (US precedent)

#365

So many ideas start to come to mind if scraping is legal. Can we start to scrape Google Search in order to bootstrap building an alternative to Google Search? Search is a really hard problem (that somebody should tackle), but if we can leverage what Google has already scraped from the web and associated with popular search terms, we can use that to help train and validate our search model. Can we scrape Reddit, Twitt…

> Can we scrape Reddit, Twitter, or Facebook in order to stand up a competing service that strips out all the ads? How are you going to pay for it? Subscription model doesn't work for search/social networks.

Ad-free Reddit would be sustainable if:

- Comments are ephemeral, expiring after two weeks (no growing storage costs)

- "Reddit Gold" helps to offset costs

- Run Wikipedia-like donation drives yearly

- Write everything in bare-metal Rust so that CPU is cheap. Likewise, make intelligent choices about schema and service design for scalability.

- Don't continue to drive unnecessary feature work (that is usually just to drive ad engagement and growth).

Re: Congrats! Web scraping is legal! (US precedent)

#366

Earlier quoted context omitted.

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want i…

> I don't believe organizations have rights, period So a group of people, joining together in a common cause, don’t have rights as members of that group? You are contradicting yourself. Organizations are simply groups of people with a shared cause. To deny rights to the organization, you necessarily have to deny personal rights. Example: John and Sam form an organization for comic book collecting. They have a secret…

> Organizations are simply groups of people with a shared cause.

No, they're not. As soon as you have two people in an organization, you've got two different causes. The shared elements of those causes allow them to collaborate, but each of them has slightly different views of what they're working toward. And even when you have a very small, well-specified goal, every individual in the organization has different levels of investment, and other values and boundaries they're not willing to cross to achieve that goal. And the larger organizations get, the wider the variety of disparate goals that can occur within the organization, because individuals in the organization may not even interact directly with one another.

Example: John and Sam work for Facebook. John and Sam want to make money to feed themselves and their families, and don't want to have PTSD. But Mark Zuckerberg wants John and Sam to look at an endless stream of horrific PTSD-inducing images so that he can maintain the reputation of the Facebook platform and get incredibly rich. Where is the shared cause here, exactly?

> To deny rights to the organization, you necessarily have to deny personal rights.

Just because an organization doesn't have rights doesn't mean we have to go out of our way to take away their rights.

> Example: John and Sam form an organization for comic book collecting. They have a secret meeting to agree on a price to offer a potential seller of a rare comic book. Since John and Sam, Inc. “Have no rights” the seller is allowed to sit in the room during their meeting. However this violates the right of Sam to freely associate with John and his right to not have to associate with the interloper. It also violates the privacy of San and John as individuals since their private conversation — even as members of their two-man organization are now being shared with anyone who wants to sit in since, in this scenario, their organization doesn’t have rights.

It sounds like you've figured out that John and Sam, as individuals, each have a right to have a private conversation. John and Sam, Inc. doesn't have rights, but that doesn't suddenly remove John and Sam's individual rights.

The entire point of organizational transparency is to protect the rights of individuals, so obviously if organizational transparency would violate the rights of individuals, the individual rights to privacy supercede the need for organizational transparency.

It's telling that your example of an organization has two people in it. At a small scale, organizations tend to protect the rights of the individuals in the organization fairly well. It's at larger scales that the non-rights of an organization come in conflict more often with the rights of individuals.

Re: Congrats! Web scraping is legal! (US precedent)

#367
post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

Yeah, I have nothing at all against scraping per se, it's more about the huge bot traffic the commercial scrapers generate, which would ALSO be fine, but 1. it can be hard to tell scrapers from malicious DDOS bots sometimes and 2.) the person being scraped literally pays for that scraping traffic.

So, apply DOS/DDOS mitigations? You need to have those anyway, so you might as well use them for this too.

Re: Congrats! Web scraping is legal! (US precedent)

#368
post #128

Earlier quoted context omitted.

One of my clients is involved in property tax collection and reporting. Property Tax records are public info, and their website allows looking up the records for any property without a login. However, the data behind this website it the _source_ of the public records, and not the public records themselves (which would be local government databases). For years now we've been in an arms race with someone using a botnet…

Would rate limiting be a viable solution?

No. They botnets works through multiple regions on multiple cloud providers - that's how they achieve such high throughput. For any single IP address, the load is reasonable, but for the whole botnet it's absurd.

Currently bot traffic accounts for 2/3 of my load, meaning that the cost of providing my service is 3x what it would be without these persistent bots.

Re: Congrats! Web scraping is legal! (US precedent)

#369

Earlier quoted context omitted.

I'm going to assume you're asking in good faith and try to address the confusion here. The human does get rights, the organization doesn't. In some cases, believing that humans have rights and believing that organizations have rights might lead one to the same action. In those cases, I'd take the action. I wouldn't want to violate a human's rights out of some vindictive dislike of organizations: that's not the point.…

Are contracts allowed in your worldview? Contracts must be signed by individuals, but when they act as representatives of a company, they are legally binding for that company. If they only have individual rights, then all individuals who didn't physically sign a contract cannot be held to it.

Correct.

At a small scale, it makes sense for people to appoint another person whom they trust to represent them in negotiation. But in a lot of cases, that's not how representatives are chosen. Particularly in corporations, the leadership of a corporation was not chosen by the employees to represent them, and in fact often doesn't even have the best interest of the employees in mind. A lot of the largest problems in our society arise from this fact.

Consider the case of a company that agrees to sell to a larger corporation, under the condition that they lay off half their workforce in advance of the sale. Surely we can agree that the laid-off workers were not fairly represented by the person signing the contract.

One could argue that the workers agreed to give up some of their rights in their employment contract, but I'd argue that they did so under duress: their option is sign the contract and work for the company, or starve and let their families starve. Sure, they can go work for another company, but other companies will require them to similarly sign away their rights.

This shouldn't be taken as a recommendation to blithely break contract law. Corporations don't have rights, but they do have power, and it would be unwise to behave as if they can't make your life miserable if you decide to cross them.

Re: Congrats! Web scraping is legal! (US precedent)

#370

Earlier quoted context omitted.

> People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. That's a complete misconception. Of course you can find manufacture inconsistent ideologies if you combine ideas from different people, but I think you'd have a difficult time finding one person who believes what you just described. What I want i…

You see this inconsistency in individual people all the time. Like how almost everyone wants to be found easily on LinkedIn, Twitter, etc, so they add a public profile photo. Then they freak out because someone starts scraping these photos to build a facial recognition model.

I think that comes from ignorance of the implications of uploading their photo, not from an inconsistency in what they want.
Post reply on HN