Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LinkedIn: It’s illegal to scrape our website without permission

arstechnica.com

Post reply on HN
Print view View original

121–130 of 303 posts

Re: LinkedIn: It’s illegal to scrape our website without permission

#121

Post by Chris2048 »

wastedhours wrote: post #54
aphexbr wrote: post #41

Earlier quoted context omitted.

The bike example is a very poor analogy - the data isn't removed from LinkedIn, it's merely copied. It doesn't matter how many times the site is scraped, the data is unchanged and still available. A better analogy would be me taking a photo of the bike while walking past. It shouldn't matter how many times you tell me to stop, if it's on public property you can't really stop me from taking the photo. If you have an i…

Ok, if I left the prototype for a new bike on the pavement, and you came along and 3D printed an exact replica. Sure, the original still exists, but you just violated trademark/registered design laws because it was there. It's not the same as a photo because a photo of a bike doesn't give you the same value as the actual bike, whereas scraping the content of a webpage does give you the same value. Perhaps my analogy…

> you just violated trademark/registered design laws

If it has those. The data on LI (other people's employment histories) is not its own IP.

Re: LinkedIn: It’s illegal to scrape our website without permission

#122

Post by Chris2048 »

magnetic wrote: post #94
williamle8300 wrote: post #81

Earlier quoted context omitted.

Actually, no. It's like taking pictures of the paintings from the street, and reselling those pictures. If they don't like that... that the painting down. Simple.

Sounds like an Intellectual Property violation to me. You don't have to hide your IP in order to protect it: there are laws that deal with this.

What IP is contained within LI?

Re: LinkedIn: It’s illegal to scrape our website without permission

#123

Post by debacle »

mywittyname wrote: post #118
debacle wrote: post #113

Earlier quoted context omitted.

> Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it. This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.

They are within their right to require authorization to their content, yes, but they don't do that. They make it public and allow some companies to scrape that content and resell it for profit (like Google), but while restricting other companies from doing the same. If they don't want their data to be public, then they shouldn't make it public. They could require authorization to view any content on the site and solv…

What leads you to believe that LinkedIn's data is public?

Re: LinkedIn: It’s illegal to scrape our website without permission

#124

Post by whatnotests »

yourapostasy wrote: post #103
whatnotests wrote: post #97

Earlier quoted context omitted.

LinkedIn gutted their API a couple years ago and now the information they took out has been moved to a private API If your business is in the recruiter space, expect to have your API keys revoked and to receive a cease and desist letter as well.

This sounds like the setup to an up-spiraling arms race with "dark scrapers". I'm guessing behind-the-scenes, LinkedIn figures they can out-spend the extralegal scrapers, and likely considers their efforts will deliver halo effects to the rest of Microsoft. It would be educational to hear how LinkedIn plans to take down bot-net-based scraping that uses deep learning to identify patterns that successfully mimic human…

If the goal of such an operation were to effectively create an alternative to LinkedIn, along the likes of other "claim your listing" sites, then this could be a worthwhile cause.

Re: LinkedIn: It’s illegal to scrape our website without permission

#125

Post by Chris2048 »

wccrawford wrote: post #5

>One plausible reading of the law—the one LinkedIn is advocating—is that once a website operator asks you to stop accessing its site, you commit a crime if you don't comply. I'm fine with that. But I'm also fine if they do the like the article suggests and require anything non-scrapable to be behind an account prompt, even if everyone with a account can access it. I don't think it's fair to make Linked In foot the bi…

> They shouldn't have to serve that content to people who aren't actually their users.

And insurance companies only want to serve people who won't claim etc. LI is free to force users to usage agreements, but why should third-parties (taxpayers) foot the bill for enforcing them?

Re: LinkedIn: It’s illegal to scrape our website without permission

#127

Post by kharms »

yourapostasy wrote: post #103
whatnotests wrote: post #97

Earlier quoted context omitted.

LinkedIn gutted their API a couple years ago and now the information they took out has been moved to a private API If your business is in the recruiter space, expect to have your API keys revoked and to receive a cease and desist letter as well.

This sounds like the setup to an up-spiraling arms race with "dark scrapers". I'm guessing behind-the-scenes, LinkedIn figures they can out-spend the extralegal scrapers, and likely considers their efforts will deliver halo effects to the rest of Microsoft. It would be educational to hear how LinkedIn plans to take down bot-net-based scraping that uses deep learning to identify patterns that successfully mimic human…

I'm skeptical of the potential of dark scrapers at scale. You'd need to simulate too much human behavior to be unidentifiable, and humans are slow.

You would need real-looking bot accounts that you'll use to scrape. You'd need a realistically randomized rate limit, sampling from some distribution conditional on the type of the source page. You'd need realistic mouse/keyboard movements. Realistic hours of operations. Can't be scraping at 4AM and 4PM, and all of the hours in-between. Occasional noise operations, such as searching for a job, or getting salary estimates. You'd be geographically constrained. You wouldn't want your bot from Boston to be looking at too many individuals in Houston (regularly). Maybe you'd use a Markov chain to have the bot make decisions? I doubt the blackhats would have good training data for a neural net. You'd need tens of thousands of these bots to cover the linkedin user base in reasonable time (say, once every week on average), and these bots would have to either overlap or seriously underlap on who they cover.

Best use case would scraper-API that you can use to look up batches of specific people, with your bots looking at others only to look realistic.

(Or maybe not? It's a fun question, but I know fuck all about this. Not my area of expertise.)

Re: LinkedIn: It’s illegal to scrape our website without permission

#128

Post by bostonpete »

lightbyte wrote: post #107
openmosix wrote: post #102

Earlier quoted context omitted.

The type of data being discussed here (factual data about people) cannot be copyrighted - i.e. the fact John Doe is a Software Engineer for ACME Inc is not copyrightable.

A better example I think is walking into a store and writing down what they price everything as, then selling aggregate pricing data to people. I can't see any reason that would be illegal.

Would likely be illegal if you continued to do after the store asked you to leave...

Re: LinkedIn: It’s illegal to scrape our website without permission

#129

Post by [deleted] »

magnetic wrote: post #94
williamle8300 wrote: post #81

Earlier quoted context omitted.

Actually, no. It's like taking pictures of the paintings from the street, and reselling those pictures. If they don't like that... that the painting down. Simple.

Sounds like an Intellectual Property violation to me. You don't have to hide your IP in order to protect it: there are laws that deal with this.

[deleted]

Re: LinkedIn: It’s illegal to scrape our website without permission

#130

Post by wslh »

pinaceae wrote: post #117
wslh wrote: post #50

Earlier quoted context omitted.

This is not what they are selling with the premium offering and this why they are lying. They are offering a way to improve selling your products and services but without any technical innovation beyond messaging.

LinkedIn has two main functions: 1., Self-updating rolodex for sales people. 2., Recruitment tool. It works for both, no real competitor in sight due to the massive network effect. Some niche networks are doing ok, like Xing in DACH region. Not aware of anything special in China, guess everyone is on WeChat anyhow.

This is not what they are offering, look at the premium offerings: https://premium.linkedin.com/ do you know the ROI of using LinkedIn Premium (e.g. InMail) to just contact people using a zillion of methods available, and then adding them to your LinkedIn...
Post reply on HN