I once tried to scrape http://www.diseasesdatabase.com by all means, but failed so hard I can't even tell this story without embarrassment.
They have all kinds of blocks in place.
LinkedIn: It’s illegal to scrape our website without permission
111–120 of 303 posts
Re: LinkedIn: It’s illegal to scrape our website without permission
#112If you post it on a public network, it is defacto, public. If you don't want it scraped, take it down, or put it behind a login. If the user provides the login to a scraper, then the scraper has permission.
What if the user is provided with permission to view medical records - your medical records for example? Would it be ok to scrape and sell those? What about children's school profiles and records?
Selling medical records would fall under a different law same goes for sharing credit #s cards you collect.
I think the issue is around public data here. If you made medical records available to the public you would be in trouble in many countries. Sharing those may land you in the same trouble as the website who initially posted it under that law. The person shouldn't be charged with unauthorized access because the materials are public.
Re: LinkedIn: It’s illegal to scrape our website without permission
#113I'm seeing a lot of bad analogies thrown around in this comment thread, mostly based on emotional response and/or a dislike for LinkedIn. As someone who has done a lot of scraping in the past (sometimes for good, sometimes not), the number one thing you need to respect as a scraper is that email or phone call you get saying "Stop doing that." In almost all instances, you're legally fine in the real world until you ge…
> But LinkedIn seems within their rights here. Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights. It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option…
> Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic.
But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it.
This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
Re: LinkedIn: It’s illegal to scrape our website without permission
#114Web service is not like ad on a paper or a public billboard.
If service is being scraped by other companies who makes or not makes money for it, or prevents the owner making money with it (filtering out the ads) this service is most likely to cause extra costs as bandwith and processing power. They are not free.
So you should have right to block anybody from your site as you wish!
Re: LinkedIn: It’s illegal to scrape our website without permission
#115Web service is not like ad on a paper or a public billboard. If service is being scraped by other companies who makes or not makes money for it, or prevents the owner making money with it (filtering out the ads) this service is most likely to cause extra costs as bandwith and processing power. They are not free. So you should have right to block anybody from your site as you wish!
LinkedIn is free to block the IPs of the scrapers.
Re: LinkedIn: It’s illegal to scrape our website without permission
#116"It's a fight that could determine whether an anti-hacking law can be used to curtail the use of scraping tools across the Web."
In the US. Not elsewhere. Is it possible that the centre of gravity of innovation will be somewhere, er, less litigious soon?
Reminds me of the City of London's historical approach to most forms of regulation (think: Francis Drake, privateers, the convertibility of lute strings &c).
Re: LinkedIn: It’s illegal to scrape our website without permission
#117Earlier quoted context omitted.
> LinkedIn is a scam, where the service is done for the recruiters Everyone knows that LinkedIn gets paid by recruiters rather than recruitees. But of course, that is perfectly fine. People are happy to be the product if it results in job leads.
This is not what they are selling with the premium offering and this why they are lying. They are offering a way to improve selling your products and services but without any technical innovation beyond messaging.
LinkedIn has two main functions:
1., Self-updating rolodex for sales people. 2., Recruitment tool.
It works for both, no real competitor in sight due to the massive network effect.
Some niche networks are doing ok, like Xing in DACH region. Not aware of anything special in China, guess everyone is on WeChat anyhow.
Re: LinkedIn: It’s illegal to scrape our website without permission
#118Earlier quoted context omitted.
> But LinkedIn seems within their rights here. Congress writes bad laws all the time. So LinkedIn might be within their rights, but that doesn't mean they should have those rights. It's bad for innovation to allow for selective discrimination like this. LinkedIn is perfectly happy to allow Google, Yahoo, Bing, and many, many more companies to scrape their content and use it for personal profit. Giving them the option…
> Giving them the option to sue an upstart for doing exactly the same thing as Google is unfair and oligopolistic. But it's not. LinkedIn's data is their entire business. They are within their rights to restrict access to it. This is the classic ant and grasshopper story. If HiQ wants access to the type of data they are scraping from LinkedIn, they can build that data themselves.
They are within their right to require authorization to their content, yes, but they don't do that. They make it public and allow some companies to scrape that content and resell it for profit (like Google), but while restricting other companies from doing the same.
If they don't want their data to be public, then they shouldn't make it public. They could require authorization to view any content on the site and solve this problem instantly.
Re: LinkedIn: It’s illegal to scrape our website without permission
#119Earlier quoted context omitted.
The pool has the right to kick you out, same as any website. The pool cannot call the police and charge you with a felony for misusing their resources.
Does the pool have any recourse if you proceed to bypass the ban? Do you have to re-enter to pool to bypass it, or does sending in confederates with their own badges to continue your work also bypassing the ban? How about sending in new motorized cars? The analogy is starting to break down, but I think it's still instructive for the problem of applying a simple first principles approach.
There is a legal concept known as "attractive nuisance"[1]. If I have a pool and neighborhood kids come to play and someone gets hurt, it's my fault. Even if I was away from my house and never gave permission (or explicitly forbade them from swimming), if I don't have proper access controls in place, the courts say it is too tempting for the neighbors to just come over and swim. I need to put up a locking gate to keep them out.
Likewise in some high-crime jurisdictions, if you did not lock your car you are liable for it getting stolen or broken into[2]. An unlocked car is too tempting for some people to just walk past and not take it.
I know it might sound crazy but you could make the argument that a massive pool of highly-structured and very valuable data just sitting out in the open is an attractive nuisance and steps should be taken to put it behind a locked gate. Once that requirement has been satisfied, normal trespassing laws apply.
[1] https://en.wikipedia.org/wiki/Attractive_nuisance_doctrine
[2] http://www.cbc.ca/news/canada/montreal/ndg-resident-question...
Re: LinkedIn: It’s illegal to scrape our website without permission
#120If you post it on a public network, it is defacto, public. If you don't want it scraped, take it down, or put it behind a login. If the user provides the login to a scraper, then the scraper has permission.
That's a pretty literalist, one-size fits all approach to policy. I don't think it's a good framework to use for applying ethics considerations. If I can walk near a pool, should I also be able to run? Is running anything more than faster walking? If I'm allowed to be around the pool walking with my entry ID, should I also be allowed to place my ID on a little motorized car and make it dart around the pool really fas…
The way HTTP works is there is a request and response. If LinkedIn does not want to provide a response, they don't have to. If they don't want to accept requests, then maybe the World Wide Web isn't for them.