Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Congrats! Web scraping is legal! (US precedent)

parsers.me

Post reply on HN
Print view View original

231–240 of 409 posts

Re: Congrats! Web scraping is legal! (US precedent)

#231

Post by TheCoelacanth »

Wowfunhappy wrote: post #29
kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

I want web scraping to be legal—but, is it really contradictory to say "I want this data to be accessible to real humans only"? Any person can post on Hacker News. However, if someone made a bot to post to Hacker News, I think most of us would be pretty upset.

I don't really care if the comments are from bots, per se. I care if they are quality comments or not. Whether or not the comments are from bots is just a proxy for whether or not they are actually good.

Re: Congrats! Web scraping is legal! (US precedent)

#232

Post by fortran77 »

Why do we have this editorialized title, @Dang? Why not the actual, neutral, title: "US court fully legalized website scraping and technically prohibited it"

BTW: This title is still not factual, but at least it's more neutral. It seems to be an injunction, which doesn't really mean "fully legalized" but what do I know....

Re: Congrats! Web scraping is legal! (US precedent)

#233

Post by Dayshine »

derefr wrote: post #218
Dayshine wrote: post #206

Earlier quoted context omitted.

Well, my name, my job title, my employer, my job history. These are all my information, and selling them to marketing companies is definitely not archiving. Would you be OK with a company scraping your blog and selling it?

> Would you be OK with a company scraping your blog and selling it? Selling it how? If they put my blog posts in a book and try to sell that book, that’s copyright infringement. If they put my blog posts in an ML model corpus to train a translation service, and they then charge pay-per-use access to the resulting service... I don’t think I’d care, nor do I think there’s anything morally or legally wrong with that. If…

Why can't I have terms on my website that say how you can use my information?

Examples where this is allowed:

- Images/media (Creative commons)

- Code (Open source licenses)

You say it isn't allowed for:

- Personal data

Unless I'm misunderstanding your philosophy (which seems to say copyright is OK, but public information must be public to all): You believe that it's morally OK for me to prevent a company selling my book, but not morally OK for me to prevent a company selling my name, job title and employer as a marketing bundle?

Edit: An aside, it's really confusing that you seem to be editing your previous replies minutes after I responded. I thought HN only let users edit during the "no replies" period?

Re: Congrats! Web scraping is legal! (US precedent)

#234

Post by LameRubberDucky »

jakelazaroff wrote: post #120
kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

I want to be able to use LinkedIn to network with colleagues and people in my industry. If someone wants to scrape my profile to make a report on industry trends, I’m fine with it. What I don’t want is hiQ vacuuming up my data so they can snitch to my employer if they think I’m job hunting. How is this a paradox? Tech — the web in particular — is supposed to be an equalizing force, but HiQ is clearly trying to give m…

Wow, I just looked up what hiQ does and have to say it's pretty scummy in my opinion. Why do people create stuff like this? Don't they know it will likely come back to bite them one day?

For reference:

"There is more information about your employees outside the walls of your organization than inside it. hiQ curates and leverages this public data to drive employee-positive actions.

Our machine learning-based SaaS platform provides flight risks and skill footprints of enterprise organizations, allowing HR teams to make better, more reliable people decisions."

Re: Congrats! Web scraping is legal! (US precedent)

#235

Post by zipzipzip101 »

powrtoch wrote: post #38

"HiQ only takes information from public LinkedIn profiles. By definition, any member of the public has the right to access this information. Most importantly, the appeals court also upheld a lower court ruling that prohibits LinkedIn from interfering with hiQ’s web scraping of its site." Surely I'm not reading this correctly. This would seem to suggest that websites are not legally allowed to prevent bots from crawli…

I have to scrape linkedin sometimes for work and it's hard. Normal Python scrapy setups won't work. Linkedin has all sorts of blockers in place. They'll give you 500 errors, log you out if you view too many pages, etc.

Headless chrome + only hitting 50 pages max per session is the key.

Re: Congrats! Web scraping is legal! (US precedent)

#236

Post by 6gvONxR4sf7o »

derefr wrote: post #143
yodon wrote: post #26

Earlier quoted context omitted.

> People want their data to be public People don't want their data to be public. People want other people's data to be public. One's own data everyone thinks should be private and tightly controlled. This applies to people and businesses equally.

In this case, LinkedIn users kind of do want their “public profiles” to be public. They’re online CVs; by definition, if you make one, your goal is to get it into the hands of anyone who asks for it! LinkedIn, likewise, has built its business model on an implicit contract with its users that it’s going to show their CV to anyone who asks for it. I think LinkedIn users would be surprised that LinkedIn doesn’t let bots…

I have a linkedin so that I can point people at it. I also want human recruiters who have actually read the thing to send me relevant jobs. If my profile ended up affecting my credit report, I'd be pissed. I expect you would be too.

People put data places for specific purposes (to show recruiters) and want the ability to limit use to that purpose. How that's accomplished is just a technicality most people don't care about.

Re: Congrats! Web scraping is legal! (US precedent)

#237

Post by orf »

Tactic wrote: post #213
kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

The issue here for some, if not many, is a matter of scale. It is one thing if an end-user, whom I am trying to service, comes to my site and gets my publicly available data. Maybe I monetize with ads, maybe not. It doesn't matter, that is the audience I am trying to service, regardless of size. But when you scrape it my load goes up dramatically. A load I have to pay for. It is analogous to the privacy debates going…

Provide an API for public data to reduce the costs associated with rendering a full blown page, and deliver just the information needed.

Re: Congrats! Web scraping is legal! (US precedent)

#239

Post by jayd16 »

levosmetalo wrote: post #73
kabacha wrote: post #11

The toxicity towards web-scraping is really what makes me lose hope in the current web. People want their data to be public and all of the benefits that comes with public data but then they want to chose who gets to see it - it's a complete and utter paradox. This precedent doesn't really mean much but is definitely step in the right direction.

Sure. My data is still my data, and if I publish it on my platform for free, that still shouldn't automatically give you the right to copy the data and provide on your platform. It's basically the same as a TV broadcasting a film for free, and then going after you legally if you recorded that film and uploaded it to your website.

This is a bad analogy as scraped but copyrighted works are still protected by copyright. Whats in question is whether you should have exclusive rights to information you have shared but did not copyright. Seems people also think this noncopyrighted data should also be protected even if its munged and added to an original work by the scraper.

Re: Congrats! Web scraping is legal! (US precedent)

#240

Post by kerkeslager »

whatshisface wrote: post #156
kerkeslager wrote: post #132

Earlier quoted context omitted.

I'm going to assume you're asking in good faith and try to address the confusion here. The human does get rights, the organization doesn't. In some cases, believing that humans have rights and believing that organizations have rights might lead one to the same action. In those cases, I'd take the action. I wouldn't want to violate a human's rights out of some vindictive dislike of organizations: that's not the point.…

Let's say that individual humans have the right to keep secrets. Let's also say that they have the right to keep secrets with their associates, and to tell them to who they please. Now, doesn't that make it legal for a group of people to keep secrets about you ? What about selling them? I just don't see what doing away with the legal fiction of corporate personage would do about Facebook.

It may not be your intent, but you're using some very vague, inapplicable terminology to make some screwed up behavior sound normal.

If you can tell secrets to who you please and sell them on the internet, they aren't secrets. Somewhere in the middle of what you're saying, the secrets stopped being secrets, but you kept using the word as if it still applied.

Facebook isn't a group of associates trading anecdotes about their friends: the server guy has never met Mark Zuckerberg, and they are not "associates" in any meaningful way. They're not friends, or even really allies: Facebook certainly has shown inconsistent concern for the well-being of its workers. So let's also drop the "associates" terminology: these aren't "associates", they're employers and employees. Employees aren't acting as individual humans on their own behalf, they're acting on behalf of an organization.

Putting aside the rights conversation for a second, let me ask you a question: if you tell your friend a secret in confidence, and they turn around and sell it to anyone on the internet who will pay a low fee, that would be pretty screwed up, no? We don't even have to talk about rights here: this is just screwed up behavior, regardless of the rights conversation.

Post reply on HN