Sorry but I side with Facebook, a freely available public graph of millions of users could have been used for re-identification attacks.
Frankly you should never share your friends list publicly.
How I got sued by Facebook (2010)
11–20 of 39 posts
Re: How I got sued by Facebook (2010)
#12Someone convince me what facebook said here was wrong. I don't think robots.txt gives you a license to do whatever you want with web content. If it did wouldn't robots.txt effectively put everything into the public domain?
Re: How I got sued by Facebook (2010)
#13So...anyone have a mirror of the data?
google for "fbnames".
Re: How I got sued by Facebook (2010)
#14Lawsuit nastiness aside, there's an interesting and important legal-technical question that this exposes: how should websites specify acceptable uses of crawled data and other fine-grained restrictions in a machine-readable form.
Motivated by this incident, I got together with Pete (the author/victim) to write a piece on "The Need to Reboot Robots.txt" [1] but it went nowhere.
Any suggestions on how to give our proposal legs would be much appreciated.
[1] http://33bits.org/2010/12/05/web-crawlers-privacy-reboot-rob...
Re: How I got sued by Facebook (2010)
#15Someone convince me what facebook said here was wrong. I don't think robots.txt gives you a license to do whatever you want with web content. If it did wouldn't robots.txt effectively put everything into the public domain?
Google can mine the data and do whatever they want (and I don't doubt for a second that they run analysis on it), but this guy can't?
Facebook wants to have their cake and eat it to. They want free Google publicity but god forbid some dude starts downloading pages for research. It's legal, but it's wrong.
Re: How I got sued by Facebook (2010)
#16I notice they have updated their robots.txt to only allow user agents they have approved.
Re: How I got sued by Facebook (2010)
#17You might care to read the extensive discussion from when this was posted 11 months ago:
Re: How I got sued by Facebook (2010)
#18Lawsuit nastiness aside, there's an interesting and important legal-technical question that this exposes: how should websites specify acceptable uses of crawled data and other fine-grained restrictions in a machine-readable form. Motivated by this incident, I got together with Pete (the author/victim) to write a piece on "The Need to Reboot Robots.txt" [1] but it went nowhere. Any suggestions on how to give our propo…
[deleted]
Re: How I got sued by Facebook (2010)
#19Lawsuit nastiness aside, there's an interesting and important legal-technical question that this exposes: how should websites specify acceptable uses of crawled data and other fine-grained restrictions in a machine-readable form. Motivated by this incident, I got together with Pete (the author/victim) to write a piece on "The Need to Reboot Robots.txt" [1] but it went nowhere. Any suggestions on how to give our propo…
You can set the SyndicationRight directive for OpenSearch.
"Contains a value that indicates the degree to which the search results provided by this search engine can be queried, displayed, and redistributed."
The default is "open" meaning: - The search client may request search results. - The search client may display the search results to end users. - The search client may send the search results to other search clients.
http://www.opensearch.org/Specifications/OpenSearch/1.1#The_...
That would give you more fine-grained control over what search agents do with your data. I don't know how broad the support and adherence is to the OpenSearch spec (IMDB uses it).
Re: How I got sued by Facebook (2010)
#20Someone convince me what facebook said here was wrong. I don't think robots.txt gives you a license to do whatever you want with web content. If it did wouldn't robots.txt effectively put everything into the public domain?
Google can mine the data and do whatever they want (and I don't doubt for a second that they run analysis on it), but this guy can't? Facebook wants to have their cake and eat it to. They want free Google publicity but god forbid some dude starts downloading pages for research. It's legal, but it's wrong.
I'm really only interested in the legal question. And I genuinely would like to be convinced that the legal system would allow scraping like this. I just don't see it.