Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LinkedIn: It’s illegal to scrape our website without permission

arstechnica.com

Post reply on HN
Print view View original

231–240 of 303 posts

Re: LinkedIn: It’s illegal to scrape our website without permission

#231

Post by hackits »

pdabbadabba wrote: post #150
briandear wrote: post #146

Earlier quoted context omitted.

I wasn’t aware that robots.txt had any legal meaning. Am I wrong?

> Am I wrong? I hate to be the bearer of bad new but...maybe. A reading of the Computer Fraud and Abuse Act could make robots.txt legally enforceable. And given the government's approach to CFAA cases a very aggressive interpretation, under the right circumstances (for example, when it provides evidence that the scraper knew that scrapint was not authorized), seems like a real possibility. Among the many other things…

I wouldn't be too fast to jump to conclusion robots.txt legally enforceable. You would need to cite prior case law's. Without any case law's it make decision on a law error prone at best.

Re: LinkedIn: It’s illegal to scrape our website without permission

#232

Post by ChuckMcM »

__jal wrote: post #229
ChuckMcM wrote: post #213

Earlier quoted context omitted.

If your robots.txt file is /allow then you did. If you have no robots.txt file then it's an open question. If you put a /deny into your robots.txt file Google will stop scraping your site. The implicit contract is that you let them scrape because you want to show up in their search results which will send you traffic. If you don't care about Google traffic then set /deny in your robots.txt and get back the bandwidth…

> If you have no robots.txt file then it's an open question. Only for definitions of explicit I must be unfamiliar with. If the presence of a robots.txt makes one's intent for a given resource explicit one way or the other, the lack of one (and the lack of some communication in some other channel) must mean there is no explicit permission.

That is correct, for what it was worth IBM's legal team came down on the side of 'assume deny' and Google was (at the time I was there) 'assume allow.'

Re: LinkedIn: It’s illegal to scrape our website without permission

#233

Post by jonbarker »

OkGoDoIt wrote: post #144
5trokerac3 wrote: post #141

Unpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't have a right to stop scraping after the fact, but they have the right to take technical steps to stop scrapers from accessing their site.

LinkedIn takes plenty of technical precautions to block scraping. I’ve built bots that scrape them in the past, it’s surprisingly difficult as LinkedIn is very good at determining you’re a bot and blocking you. So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. Whether or not this is illegal is up to the courts to determine, and…

Don't the other social networks throttle http requests from certain dubious ip addresses? I think they all do this.

Re: LinkedIn: It’s illegal to scrape our website without permission

#234

Post by vertex-four »

OkGoDoIt wrote: post #144
5trokerac3 wrote: post #141

Unpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't have a right to stop scraping after the fact, but they have the right to take technical steps to stop scrapers from accessing their site.

LinkedIn takes plenty of technical precautions to block scraping. I’ve built bots that scrape them in the past, it’s surprisingly difficult as LinkedIn is very good at determining you’re a bot and blocking you. So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. Whether or not this is illegal is up to the courts to determine, and…

> a sleazy website that convinces people to willingly provide personal information which they then turn around and sell at ridiculously high prices

I think you just defined a social network, unfortunately.

Re: LinkedIn: It’s illegal to scrape our website without permission

#235

Post by [deleted] »

ChuckMcM wrote: post #177
5trokerac3 wrote: post #141

Unpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't have a right to stop scraping after the fact, but they have the right to take technical steps to stop scrapers from accessing their site.

I don't think you've characterized this accurately. When you make an HTTP request to LinkedIn you are accessing their service. There is a long history of this relationship, you plug your house into the sewer line and you connect to the sewer service. You connect to the power pole and connect to the electricity service. You connect to the telephone pole and connect to the telephone service. Every service has "terms of…

[deleted]

Re: LinkedIn: It’s illegal to scrape our website without permission

#236

Post by aleem »

ChuckMcM wrote: post #177
5trokerac3 wrote: post #141

Unpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't have a right to stop scraping after the fact, but they have the right to take technical steps to stop scrapers from accessing their site.

I don't think you've characterized this accurately. When you make an HTTP request to LinkedIn you are accessing their service. There is a long history of this relationship, you plug your house into the sewer line and you connect to the sewer service. You connect to the power pole and connect to the electricity service. You connect to the telephone pole and connect to the telephone service. Every service has "terms of…

The scenario is a bit more nuanced though and creeps into Internet freedom.

- Reminds me of CraigsList vs PadMapper[1]. In that scenario I side with CL -- it was right to block PM. PM or others should not be allowed to build a new UI on top of CL because CL was the one that put in years of effort of nurturing its listings, its network, building brand equity and taking associated risks and costs.

- As others have highlighted, the data is publicly accessibly and there is no agreement the scraper/crawler is bound by. The agreement is between the LinkedIn user and LinkedIn. The scraper is connected to the Internet pipe crawling the Internet freely as it wants. It's not reproducing the data anywhere so copyright should not be an issue.

- What if a scraper didn't scrape LinkedIn but just the Google or Archive.org cached versions and read those instead? It would not be pressuring LinkedIn server resources in this case.

- What if all of my employees allow me to scrape their LinkedIn data? Can I scrape all of their info? Can LinkedIn stop me from doing that (In the case of Facebook vs Power Ventures, the answer is that LinkedIn would be able to prevent this behaviour).

- Who owns the data? Medium.com doesn't own the posts. LinkedIn doesn't own the CVs.

  [1]: https://news.ycombinator.com/item?id=4286325

Re: LinkedIn: It’s illegal to scrape our website without permission

#237

Post by jboy55 »

havetocharge wrote: post #220
jboy55 wrote: post #205

Earlier quoted context omitted.

True, but in making the request, you will provide information on who is making that request. If you say, "I am a bot!", and they grant you permission, your request is legal. But if you say, 'I am NOT a bot', like spoofing a browser's user agent string, but you are a bot, then you are requesting access under a pretense, in order to circumvent their terms of service. Kinda feels morally wrong, and illegal.

There isn't a universal 'I'm a bot' setting. There are user agent conventions, but they are hardly standard. Your point works in theory, but it's not something one can just implement and be reasonably confident that they won't be scraped.

No, it won't prevent being scraped. That's not the point I was making.

The point is, the scraper would have to hide their intentions and identity, which removes any claim they are being 'honest' in their intentions and not trying to circumvent the provider of the services efforts to prevent scraping.

Re: LinkedIn: It’s illegal to scrape our website without permission

#238

Post by lgas »

betenoire wrote: post #222
lgas wrote: post #191

Earlier quoted context omitted.

> So it’s hard to argue that a service which is scraping LinkedIn is doing it without knowledge that they are going against LinkedIn’s wishes. As far as I can tell, no one has made that argument, so I'm not sure why you feel the need to rebut it. I think it all pretty much boils down to this quote from the article: > LinkedIn's position disturbs Orin Kerr, a legal scholar at George Washington University. "You can't p…

> no one has made that argument, so I'm not sure why you feel the need to rebut it. The title is "It’s illegal to scrape our website without permission". So that argument is implied in the headline, at least. As for Orin Kerr, I'm sure he'd agree that there are private parts of the internet (my payment information being an obvious example). Just because something is deployed to the internet doesn't mean it is "publis…

I don't see anything that connects the title (written by LinkedIn) to the motivation of the people that are scraping it (not LinkedIn).

Everyone that scrapes LinkedIn (or anywhere else) either knows that they are doing it against LinkedIn's wishes or doesn't care.

Re: LinkedIn: It’s illegal to scrape our website without permission

#239

Post by flashman »

mywittyname wrote: post #110
debacle wrote: post #93

I'm seeing a lot of bad analogies thrown around in this comment thread, mostly based on emotional response and/or a dislike for LinkedIn. As someone who has done a lot of scraping in the past (sometimes for good, sometimes not), the number one thing you need to respect as a scraper is that email or phone call you get saying "Stop doing that." In almost all instances, you're legally fine in the real world until you ge…

> But LinkedIn seems within their rights here. Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights. It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option…

The question is, how would you better define hacking? To make the letter of the law match the spirit? "Exceeding authorised access" is an early attempt that predates modern internet usage. You can't just say "anything the computer lets you do, is legal" because code exploits are just a computer following the (poorly-written) instructions in its code.

Re: LinkedIn: It’s illegal to scrape our website without permission

#240

Post by geomark »

jamez1 wrote: post #216
dsacco wrote: post #10

I'm following this story with a lot of interest. I've done (and still do!) a lot of data crawling/scraping. In the past I've worked on so-called "alternative data" collection and analysis for financial forecasting. Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of…

Alternate data doesn't even need to be as sexy as satellite photos, hell you almost certainly want the data that isn't sexy, the stuff people haven't thought of because it's too boring. Alternative data vendors above all want sales, and even the funds themselves want things to show off to clients. This gives you great opportunities to look at the alternative data they aren't touching. Given this is a predominantly a…

"...tracking how many sales or users exist in databases by watching sequential IDs..."

Seems like a great application of the German tank problem [1] that was mentioned on HN the other day.

[1] https://en.wikipedia.org/wiki/German_tank_problem

Post reply on HN