It is illegal to let your browser fetch this text without my permission.
LinkedIn: It’s illegal to scrape our website without permission
211–220 of 303 posts
Re: LinkedIn: It’s illegal to scrape our website without permission
#212Earlier quoted context omitted.
> So it's fine to mess with them, even illegally, just because you don't like them? If it's not illegal, then scummy behaviour against a scummy company doesn't exactly set my moral compass off. You reap what you sow.
>moral compass The issue I see with this is in 2 part. First, any issue that comes down to "moral compass" is inherently dangerous. We can find many examples of the simple concept that what to one person is Good is to another Evil. In this case, I think Linkedin shareholders would not appreciate calls to mess with the site, or with people having trouble jobhunting because the site is going down repeatedly due to DDOS…
everything comes down to a moral compass of some kind. Your comment expects some sort of objective measurement of good and evil, but I don't see any. The law can be, and is often, in the wrong.
Most people seem to think this law (if it is held up in court) is wrong and should be changed.
You might say, oh well, we have a democratic right to change or influence our laws. But a princeton study has found no correlation between public preferences of the majority of the population and enacted policy: https://scholar.princeton.edu/sites/default/files/mgilens/fi...
From memory, the only thing that leads populations to revolt against their government is high enough food prices. Outside of that, revolts almost never happen.
What I'm trying to say is A) unjust, monopolistic, or excessive laws are probably more normal than the opposite because B) the idea that democracy means people have some weigh in lawmaking might be a myth and C) most people don't do anything about it because they only act when the very basics of their livelihood are threatened.
Your view, that some unjust or excessive laws are preferable to total chaos, seems to carry the assumption that laws are naturally benign and/or made to serve some purpose for society, therefore we should not challenge them without good reasons to do so. If the opposite is true and most laws or a high enough number of them are not just, then the fact that the vast majority of people disagrees with them is only natural.
This is a fairly long-winded way of saying that most people would say you're being downvoted because there are plenty of terrible laws that we should not acquiesce to silently.
Re: LinkedIn: It’s illegal to scrape our website without permission
#213Earlier quoted context omitted.
I worked at Google for four years, an independent search engine for 5 more after that, and at IBM after it acquired said search engine for 18 months after that. Everyone of those organizations spent many thousands of dollars on legal fees over just this question and reviewed tons of case law. Every single one of them concluded that based on how the law was written and how the web worked, there is no legal way to scra…
> there is no legal way to scrape a web site without its explicit permission to do so. Google scrapes several of my sites and I've never given Google explicit permission to do so.
If your robots.txt file is /allow then you did. If you have no robots.txt file then it's an open question. If you put a /deny into your robots.txt file Google will stop scraping your site.
The implicit contract is that you let them scrape because you want to show up in their search results which will send you traffic. If you don't care about Google traffic then set /deny in your robots.txt and get back the bandwidth you were giving them.
Re: LinkedIn: It’s illegal to scrape our website without permission
#214Earlier quoted context omitted.
* That being said, I hate LinkedIn as a company and I fully support anyone trying to mess with them So it's fine to mess with them, even illegally, just because you don't like them? * convinces people to willingly provide personal information Convincing is not forcing, and in fact, you say "willingly" yourself. Any business convinces you to willingly give them money or other value. * sell at ridiculously high prices…
I would replace the word "convinces" with "deceives". If you deceive a person into, for example, giving them access and permission to use your personal contact list however they please, then proceed to do just that, you don't have anything to complain about.
Agree. The number of times I've received LinkedIn emails from people I no longer speak to who I'm confident would have no actual inclination to connect with me is certainly in the double digits. They've all been conned into giving LinkedIn their email password, and LinkedIn is going crazy as result.
This probably happened a lot more a few years ago. Is perhaps 2FA making this harder these days?
Re: LinkedIn: It’s illegal to scrape our website without permission
#215Earlier quoted context omitted.
I disagree with your analogy. To me, the key word in "HTTP request" is request . A request is something that can be granted or not.
True, but in making the request, you will provide information on who is making that request. If you say, "I am a bot!", and they grant you permission, your request is legal. But if you say, 'I am NOT a bot', like spoofing a browser's user agent string, but you are a bot, then you are requesting access under a pretense, in order to circumvent their terms of service. Kinda feels morally wrong, and illegal.
The user agent header is not an authorization header. There is explicitly an authorization header.
Re: LinkedIn: It’s illegal to scrape our website without permission
#216I'm following this story with a lot of interest. I've done (and still do!) a lot of data crawling/scraping. In the past I've worked on so-called "alternative data" collection and analysis for financial forecasting. Without going into too much detail, a lot of hedge funds have teams constantly searching for kernels of data that can contribute some kind of signal for market movements. This data can come in the form of…
Alternate data doesn't even need to be as sexy as satellite photos, hell you almost certainly want the data that isn't sexy, the stuff people haven't thought of because it's too boring. Alternative data vendors above all want sales, and even the funds themselves want things to show off to clients. This gives you great opportunities to look at the alternative data they aren't touching.
Given this is a predominantly a web development community it always surprises me how little creativity there is in the articles on investing. Neural networks and machine learning sound cool but the reality is almost none of the readership would be able to make any money off them.
Simply tracking how many sales or users exist in databases by watching sequential IDs should be the go-to method for any web developer trying to get an edge. I would have expected HN to have articles where people are getting creative on that, ie trying to use measures of entropy on usernames to get rough subscriber numbers etc.
Even plain scraping of prices etc, is often full of great insight that is ignored. If a grocery store drops their prices in profitable categories against their competitors, that could be the signal about an incoming price war for an entire sector. There's a lot more information in that than social media feeds and all the other sorts of sexy data that get coverage in the media.
Re: LinkedIn: It’s illegal to scrape our website without permission
#217As far as I'm concerned, if the information is publicly visible on a site (not behind a login) and as long as the scraping doesn't cause performance issues or generate costs on the site, then it should be perfectly fine to do.
> As far as I'm concerned, if the information is publicly visible on a site then it should be perfectly fine to do. There, I fixed it.
I don't think it's okay to cause enough stress to the web host to cause delays for other visitors.
Re: LinkedIn: It’s illegal to scrape our website without permission
#218Unpopular opinion: when you make a HTTP request you're asking the server to give you information. The server has the right to say no. IMHO, LinkedIn doesn't have a right to stop scraping after the fact, but they have the right to take technical steps to stop scrapers from accessing their site.
They also state this clearly in their terms of service.
"You agree that you will not ... develop, support or use software, devices, scripts, robots, or any other means or processes (including crawlers, browser plugins and add-ons, or any other technology or manual work) to scrape the Services."
https://www.linkedin.com/legal/user-agreement
That said, this should be a breach of contract issue. It's an overreach to invoke federal fraud law.
Re: LinkedIn: It’s illegal to scrape our website without permission
#219Earlier quoted context omitted.
I worked at Google for four years, an independent search engine for 5 more after that, and at IBM after it acquired said search engine for 18 months after that. Everyone of those organizations spent many thousands of dollars on legal fees over just this question and reviewed tons of case law. Every single one of them concluded that based on how the law was written and how the web worked, there is no legal way to scra…
To the extent to which that is the case, though, it isn't due to the terms of service; and that is also a case of how you are using the data for later, which is a separate question from the scraping and collection process: it is very clear to me that a search engine is operating on the legal equivalent of thin ice, particularly with details like snippets and synthesis ;P. Whether the CFAA applies (as indicated in thi…
> it is very clear to me that a search engine is
> operating on the legal equivalent of thin ice,
For a search engine it is super clear, robots.txt is all. If you say yes explicitly, great. If you say no explicitly, that has to be honored. If you say nothing, then its up to the search engine to decide which way to interpret it, but if the site owner complains because you picked wrong you have to honor their wishes (which may include destroying any cached data as well).
PadMapper, Perfect10, and the newspapers generated a ton of cases based on 'scraping a web site and using the data.' There are also about a dozen comparative shopping sites that have been dinged for the exact same issues. (look vs Amazon or vs Walmart).
Whether CFAA, DMCA, Torte law (contracts), or something else applies is constantly being discussed :-). I'm just the messenger here. I haven't found a single case that has held that the point of view of the scraper of someone else's web site should prevail. The argument that it should be allowed 'to help new businesses get off the ground' is like saying Apple should pay out some of its cash hoard as grants to startups trying to break into some business. I have yet to read anything that was sympathetic to that point of view.
Re: LinkedIn: It’s illegal to scrape our website without permission
#220Earlier quoted context omitted.
I disagree with your analogy. To me, the key word in "HTTP request" is request . A request is something that can be granted or not.
True, but in making the request, you will provide information on who is making that request. If you say, "I am a bot!", and they grant you permission, your request is legal. But if you say, 'I am NOT a bot', like spoofing a browser's user agent string, but you are a bot, then you are requesting access under a pretense, in order to circumvent their terms of service. Kinda feels morally wrong, and illegal.
There isn't a universal 'I'm a bot' setting. There are user agent conventions, but they are hardly standard. Your point works in theory, but it's not something one can just implement and be reasonably confident that they won't be scraped.