Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LinkedIn: It’s illegal to scrape our website without permission

arstechnica.com

Post reply on HN
Print view View original

161–170 of 303 posts

Re: LinkedIn: It’s illegal to scrape our website without permission

#161

Post by int_19h »

debacle wrote: post #113
mywittyname wrote: post #110

Earlier quoted context omitted.

> But LinkedIn seems within their rights here. Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights. It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option…

> Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it. This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.

> But it's not. LinkedIn's data is their entire business.

"There has grown in the minds of certain groups in this country the idea that just because a man or corporation has made a profit out of the public for a number of years, the government and the courts are charged with guaranteeing such a profit in the future, even in the face of changing circumstances and contrary to public interest. This strange doctrine is supported by neither statute or common law. Neither corporations or individuals have the right to come into court and ask that the clock of history be stopped, or turned back."

> If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.

https://en.wikipedia.org/wiki/Feist_Publications,_Inc.,_v._R....

Re: LinkedIn: It’s illegal to scrape our website without permission

#162

Post by chrshawkes »

pinaceae wrote: post #117
wslh wrote: post #50

Earlier quoted context omitted.

This is not what they are selling with the premium offering and this why they are lying. They are offering a way to improve selling your products and services but without any technical innovation beyond messaging.

LinkedIn has two main functions: 1., Self-updating rolodex for sales people. 2., Recruitment tool. It works for both, no real competitor in sight due to the massive network effect. Some niche networks are doing ok, like Xing in DACH region. Not aware of anything special in China, guess everyone is on WeChat anyhow.

Seems like Job websites like Indeed Prime and Google's new job product are serious competitors to LinkedIn.

LinkedIn is almost entirely worthless without the occasional good lead from a recruiter. I find it's mostly drowned out by the huge amounts of insignificant skin peddlers who dole out one worthless lead after another, looking for some fresh skin to peddle.

Re: LinkedIn: It’s illegal to scrape our website without permission

#163

Post by [deleted] »

keithpeter wrote: post #116

"It's a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web." In the US. Not elsewhere. Is it possible that the centre of gravity of innovation will be somewhere, er, less litigious soon? Reminds me of the City of London's historical approach to most forms of regulation (think: Francis Drake, privateers, the convertibility of lute strings &c).

[deleted]

Re: LinkedIn: It’s illegal to scrape our website without permission

#164

Post by ijafri »

int_19h wrote: post #159
ijafri wrote: post #151

I guess for LinkedIn it could matter most, because their only livelihood is CV and employer companies Data, if you steal a poor man's bread they are gonna be upset. It was dead as a social network long ago if it ever was, now their only asset is the data, so I guess their right to protect their only asset is legit.

Their desire to protect it is legit, but it doesn't necessarily give them a right to do so. A long time ago, companies that were compiling phonebooks also thought that they could use e.g. copyright to prevent others from using that compiled data for commercial gain. They were wrong, because copyright doesn't protect "mere aggregation" of data, as courts have ruled. In this case, LinkedIn is not arguing on the basis o…

Yes it's a consensus user-generarated content or facts are not liable to copy rights, I am not sure about the legality however it should not take long to prove as such.

I meant there is nothing that LinkedIn 'created' out of that users data. But it's little complex or little immoral to steal users data hosted on their infrastructure that costs them a lot.

Re: LinkedIn: It’s illegal to scrape our website without permission

#165

Post by JoblessWonder »

TomSawyer wrote: post #75
mgkimsal wrote: post #22

Earlier quoted context omitted.

possibly just filtering based on your IP?

That crossed my mind, but it'd mean they'd be willing to lock out large networks behind a NAT. I was wondering if there's something more sophisticated available.

Also, they are just locking out users that aren't signed in. If a user has a login, they are probably much more likely to err on the side of caution before locking down access in some way.

Re: LinkedIn: It’s illegal to scrape our website without permission

#166

Post by kstrauser »

debacle wrote: post #93

I'm seeing a lot of bad analogies thrown around in this comment thread, mostly based on emotional response and/or a dislike for LinkedIn. As someone who has done a lot of scraping in the past (sometimes for good, sometimes not), the number one thing you need to respect as a scraper is that email or phone call you get saying "Stop doing that." In almost all instances, you're legally fine in the real world until you ge…

Many years ago, I wrote a scraper for a certain .mil financial website. My company at the time - and its customers - had legitimate accounts on that website, but the site was horrible to use. Page loads took upwards of a minute, and there were no front-page notification systems to let you know if any of the information on the site had changed. Kind of a big deal when it was a window into the government's accounts payable system. For instance, if you didn't notice that an auditor had left a comment on an invoice, and you didn't respond to it within X hours, then the gov't could deny the invoice and you'd have to start over. The end result was our customers paying full-time employees to manually examine every single open invoice in the .mil system every single day, which just increased load on the website and made it even slower.

Enter my scraper. It copied data into a local PostgreSQL database that our customers could run reports against. A process that used to take a human 6 hours a day now took 30 error-free seconds. The scraper was even a benefit to the website as we ran it overnight during low-load conditions, and because my software was smarter than a web browser, we could retrieve the same information as a human with about 1/3 the number of web requests. Perfect, right!

Well, no. We got an angry call from the developers complaining that we were the ones making their site slow, even though 1) we measurably didn't, 2) we were lighter on resources than the humans we were replacing, and 3) we scrupulously obeyed their rate limiting requirements and erred on the side of running 10% slower than they had originally requested from their customers.

That particular problem went away when we pointed out that their shiny new website wasn't Section 508 accessibility compliant and could not be made to be without literally throwing out their entire web service and starting over, but our website was, and that if they continued to allow us to serve screen reader compatible pages to our disabled customers then there wouldn't be a need to have the .mil website shut down and a Congressional investigation launched. All parties involved decided this was a reasonable compromise.

Re: LinkedIn: It’s illegal to scrape our website without permission

#167

Post by bostonvaulter2 »

twsted wrote: post #160

What if I build a new public search engine and LinkedIn blocks me because I am not Google or Bing? They could stop potential (and much needed) competitors.

It would be interesting to make a "search engine" that returns results that can be aggregated. In the LinkedIn case that would be over a company.

Re: LinkedIn: It’s illegal to scrape our website without permission

#168

Post by nradov »

hellbanner wrote: post #34
dsacco wrote: post #10

I'm following this story with a lot of interest. I've done (and still do!) a lot of data crawling/scraping. In the past I've worked on so-called "alternative data" collection and analysis for financial forecasting. Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of…

"satellite imagery for oil tankers" - Interesting. If hedge funds floating drones above oil tankers, I'd guess they'd be accussed of corporate espionage / spying / invasion of privacy? Ok, so oil tankers are big and "in the clear". What if $TANKERCORP floats big parachute balloons above its tankers to imply "looking past these is unauthorized viewing"? Then if a HedgeFund gets a clever angle on a satellite photo.. is…

Hedge fund researchers have also chartered private airplanes to fly over oil storage facilities and use infrared cameras to check tank levels. In the USA at least this is completely legal as long as they observe regular FAA flight rules. For the oil market as a whole this is a good thing since it helps price discovery.

It's not practical or safe to put a huge parachute or balloon over a tanker to block overhead imagery. Any sailor can tell you it just wouldn't work.

Re: LinkedIn: It’s illegal to scrape our website without permission

#169

Post by komali2 »

vkou wrote: post #156
DeusExMachina wrote: post #149

Earlier quoted context omitted.

* That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them So it's fine to mess with them, even illegally, just because you don't like them? * convinces people to willingly provide personal information Convincing is not forcing, and in fact, you say "willingly" yourself. Any business convinces you to willingly give them money or other value. * sell at ridiculously high prices…

> So it's fine to mess with them, even illegally, just because you don't like them? If it's not illegal, then scummy behaviour against a scummy company doesn't exactly set my moral compass off. You reap what you sow.

>moral compass

The issue I see with this is in 2 part.

First, any issue that comes down to "moral compass" is inherently dangerous. We can find many examples of the simple concept that what to one person is Good is to another Evil. In this case, I think Linkedin shareholders would not appreciate calls to mess with the site, or with people having trouble jobhunting because the site is going down repeatedly due to DDOS or whatever.

The second is that these kinds of calls to action (linkedin sucks, fuck with it) smell like vigilantism to me, and while Batman is my favorite hero (I really only lift because I kinda sorta wanna be batman), vigilantism doesn't contribute to a stable society. Rule of law works better than the chaos of multiple agents enforcing their own moral code as law.

EDIT: I'm happy to be downvoted if I'm saying something stupid, but while doing so I would very much appreciate a quick comment as to why I'm wrong so I can improve my knowledge.

Re: LinkedIn: It’s illegal to scrape our website without permission

#170

Post by crazypyro »

DeusExMachina wrote: post #149
OkGoDoIt wrote: post #144

Earlier quoted context omitted.

LinkedIn takes plenty of technical precautions to block scraping. I’ve built bots that scrape them in the past, it’s surprisingly difficult as LinkedIn is very good at determining you’re a bot and blocking you. So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. Whether or not this is illegal is up to the courts to determine, and…

* That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them So it's fine to mess with them, even illegally, just because you don't like them? * convinces people to willingly provide personal information Convincing is not forcing, and in fact, you say "willingly" yourself. Any business convinces you to willingly give them money or other value. * sell at ridiculously high prices…

>So it's fine to mess with them, even illegally, just because you don't like them?

The previous commenter never mentioned messing with them illegally and disagrees with the analysis that this scraping is illegal. He said he hopes the courts do not rule this is illegal. Pretty disingenuous to start a reply like that...

Many of your other points are not really giving the previous commenter any charity at all.

For example,

>For what definition of legitimately? Yours? Since it's their business, they can define what is a legitimate free use and what can be a paid one.

He was literally trying to provide an example of where their scraping protections can be appear to be overzealous to casual users, not arguing whether those users are legitimate or not.

What is the premise of your argument? To me, it seems you are simply trying to defend LinkedIn's business practices and legal pursuits, rather than discussing anything about the legality of scraping or the specifics of LinkedIn's anti-scraping implementations.

Post reply on HN