Good News: Craigslist drops exclusive license to your posts

From Squid/Varnish style proxies, to paid CDN's, to free CDNs. If you upload content to the web, and you do not put it behind access controls like a password, it gets copied.

It's the nature of the web.

Many sites have built businesses around caching. Not taking content from search engine caches, but caching content that users request through their proxy. The users have no idea they are using a proxy. They think they are making a request to "A" site, when really their request is being passed to "B" site and "A" site is caching the result. Then other users come along and request the same content from "A" site and they think, "Wow, this is fast. And 'A' site sure has lots of content."

What's strange is that guys like Craig Newmark have made a fortune running "B" site, yet they get extremely protective of their turf. They think they are _entitled_ to something. Not really. They are just lucky. Because anyone could provide the same "service". Users can get the same content (which incicentally "B" nor "A" site own) from a myriad of points on the network. The web is a vast copy machine.

There is no entitlement. These guys who have made a fortune should be thankful.

Just because you can afford lawyers does not mean you have to start suing people because you are annoyed by competitors and the lawyers advise you that indeed you can sue. They are rarely going to tell you you cannot sue. They can find something.

Copyright trolls like Righthaven are one example of egregiousness that comes to mind. But how about with the recent OpenDNS Paxfire lawsuit? Junk patent nonsense. Fighting over who gets to hijack NXDOMAINS. The suit was dropped of course. Meritless.

There are companies out there whose business consists of hijacking and manipulating traffic, whether DNS or HTTP, to show ads. And these guys think they are _entitled_ to something. Please.

They should be happy with what they get away with.

We need more Alsups and Posners. We need more adults in the room.

i.e.: http://newyork.craigslist.org/mnh/sub/3195940666.html is hosted on http://newyork.craigslist.org/

but http://newyork.craigslist.org/robots.txt makes an HTTP 302 redirect to http://www.craigslist.org/robots.txt, but http://www.craigslist.org/robots.txt is only a valid robots.txt for http://www.craigslist.org, and not for http://newyork.craigslist.org/, as the host, protocol and port of the robots.txt file is an essential part of its validity range.

so basically newyork.craigslist.org does not have any robots.txt and can be crawled as you like.

if you want more about it, see https://developers.google.com/webmasters/control-crawl-index...

  >The directives listed in the robots.txt file apply only to 
  >the host, protocol and port number where the file is hosted.

While I agree the EFF should be concerned with companies asking for exclusive licensing to user posts, because its mission is to protect individuals' digital rights, I don't see why it should concern itself with whether companies make aggregate data cacheable, scrap-able, or API-accessible to other businesses.

CL makes money by aggregating data: it attracts buyers by aggregating all local listings in one place, and it attracts sellers by aggregating customer searches for local listings in one place. IMO the EFF has no business asking CL to give its product -- aggregate data -- away for free.

Please explain, how exactly is the EFF "tap-dancing" here?

3xx (redirection)

Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404. Handling of robots.txt redirects to disallowed URLs is undefined and discouraged. Handling of logical redirects for the robots.txt file based on HTML content that returns 2xx (frames, JavaScript, or meta refresh-type redirects) is undefined and discouraged.

CDA: https://en.wikipedia.org/wiki/Section_230_of_the_Communicati...