3Taps Files Countersuit, Says Craigslist Is Anticompetitive

I am sure the orignal author does as well. This is more a question of "fair use", not robots.txt

I read the core of 3taps' counterclaim — pages 25-30 — and I recommend it. It's a good read.

Specifics of 3taps' scraping aside, I'm starting to see that this is a particular case of a general problem that also applies to Twitter, with its recent API changes. They and Craigslist each control access to an extremely large and growing volume of user-generated content that people post there with the intention of making public. Users post that content to Craigslist/Twitter primarily because each is the largest website of its type. And both sites are, increasingly, trying to preserve their control by restricting third-party access to the UGC inside them, at the cost of reduced innovation.

It's worth dwelling on the fact that large, powerful corporations can be built off the backs of user-generated content in this way. What do you do when a service like this adds a term in their TOU that you don't like? When their web UI is lacking basic functionality? Runs really slow? Doesn't work in your favorite browser? In any of these cases, you hit a big roadblock, because you don't really have a choice. The content is on Craigslist/Twitter — even though these companies did not create it.

This problem seems to be one that will recur over and over on the internet. I can see three main avenues to fight it:

1. Technological — Create decentralized services that are competitive with, and somehow eventually overtake, the centralized ones. This is what StatusNet, Tent.io and several other efforts are trying to do in the social networking space. But this is hard. Decentralization makes it much more complicated to even build a working service at all, let alone one with a consistent, smooth and intuitive user experience. And success would result in less profit, compared to solving the much easier problem of building a successful centralized service. So developers who could pull this off are strongly incentivized not to.

2. Organizational — Start nonprofit organizations that are committed to interoperability and innovation in their bylaws, and have them run centralized services. This has been done successfully at least once: Wikipedia, operated by the Wikimedia Foundation. Their nonprofit status doesn't mean everyone is happy with their management, or that Wikipedia is perfect. But they provide full data dumps, and the content is under a free license, and their Statement of Purpose (https://wikimediafoundation.org/wiki/Wikimedia_Foundation_by...) would make it hard to justify changing that. So, no matter what, you can always innovate with content on Wikipedia.

Wikipedia was the first site of its kind, so it got big the same way major commercial sites did — by having the right idea and execution at the right time. It would be hard to replicate that success in 2012 by creating a nonprofit-owned Twitter clone. But now that Wikipedia is established, it seems to work.

3. Legal — Just as there are privacy protections in the law (moreso in the EU than the US), so could there be regulatory requirements for web services over a certain size to allow some degree of interoperability.

If we don't find some kind of solution, I imagine that in 5 or 10 years, large incumbent internet sites like Twitter, Craigslist and heck, let's throw Facebook in there as well, could be the new RIAA and MPAA, using the aggregate value of their content to bludgeon competitors into submission and restrict innovation, to the detriment of the public, except that in this case, it's that very public that created the content enabling these companies to bully others in the first place. Craigslist v. 3Taps is not the end of this.

Indeed - and i believe scraping data off a publically accessible webpage (e.g., one where you do not require registration and login/password) falls under fair use, provided you do not take up more bandwidth/resources than the average user of that site.

I recall there being some precedent for this sort of fair use - something like a phone directory - the information is not copyrighted, but the arrangement and layout is. So hence, you can't just iframe a site and present it, but obtaining the data, and deriving a new work from it should fall under fair use.

If Craigslist can assert copyright protection over classified ads and can make people grant exclusive licenses to the ads they submit, then why didn't newspapers do this before the web? Shouldn't there be a nice line of precedent for Craigslist to cite in its Complaint?

I think those web developers who actually think they "own" UGC may be in for a rude awakening if something like this ever goes to trial. As a UGC site they are providing access, and that's all. There may soon come a day when users will not need third party websites in order to provide access to data they want to make public. Web developers should consider themselves lucky if they are running a UGC site and managing to make people believe they, not the users, own the content. Cease and desist letters alleging copyright infringement claims that have no legal basis (where is the case law that says anyone can copyright classified ads?) will only work for so long.

As a regular Craigslist user (buyer, seller and message board poster), my personal preference is to have material I post only accessible directly from CL. In other words, I do not want my classified ads, contact information contained therein, pictures, social banter, business ads and other information available to any external information service or website.

This is my expectation not only of CL, but of many sites. I understand that others may disagree. But, I suspect that my expectation of non-propagation of material is the default expectation for many or most sites that accept user submitted material.

If CL is somehow forced to let others scrape their content, I would welcome an option to turn on a switch that will block/hide/protect the material I contribute from external entities. On second thought, I would rather have the do not share option be the default and let me opt in to sharing everything always, share by material classification or one post at a time.