homepage pop-up: "At approximately noon on Sunday August 5th, Craigslist instructed all general search engines to stop indexing CL postings -- effectively blocking 3taps and other 3rd party use of that data from these public domain sources. We are sorry that CL has chosen this course of action and are exploring options to restore service but may be down for an extended period of time unless we or CL change practices. As soon as we know more, we will share it here and on our Twitter account."
3taps says CL blocking "all general search engines"
1–10 of 17 posts
Re: 3taps says CL blocking "all general search engines"
#2Pretty brilliant, I don't think Craiglist ever needed Google traffic at all anymore. People know to go there to buy and sell.
Re: 3taps says CL blocking "all general search engines"
#3Pretty brilliant, I don't think Craiglist ever needed Google traffic at all anymore. People know to go there to buy and sell.
Not to mention that local classifieds are not really meant to have any permanence so often times when I saw a CL listing on Google for an item I was looking for, it was already sold or the posting was deleted. I'm still looking forward to a viable Craigslist competitor though.
Re: 3taps says CL blocking "all general search engines"
#4Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
Re: 3taps says CL blocking "all general search engines"
#5Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
Re: 3taps says CL blocking "all general search engines"
#6Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
If that's true, how does 3taps know that CL is blocking search engines? Also:
$ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-Modified
Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT
Re: 3taps says CL blocking "all general search engines"
#7I don't think this is accurate. As far as I can tell, there is nothing in CL's robots.txt, meta tags, or response headers that prevents Google from indexing them. Further, requesting a CL post with the Googlebot user agent yields the same content. This only leaves the possibility that they are excluding Google via specific IP blocks, which seems unlikely. Is there something I'm missing?
Re: 3taps says CL blocking "all general search engines"
#8I don't think this is accurate. As far as I can tell, there is nothing in CL's robots.txt, meta tags, or response headers that prevents Google from indexing them. Further, requesting a CL post with the Googlebot user agent yields the same content. This only leaves the possibility that they are excluding Google via specific IP blocks, which seems unlikely. Is there something I'm missing?
i don't know, but if i understand http://3taps-statistics.qatro.com/craigslist/index.pl correctly then they are missing lots of posts. the numbers seem to be percentages.
Re: 3taps says CL blocking "all general search engines"
#9I don't think this is accurate. As far as I can tell, there is nothing in CL's robots.txt, meta tags, or response headers that prevents Google from indexing them. Further, requesting a CL post with the Googlebot user agent yields the same content. This only leaves the possibility that they are excluding Google via specific IP blocks, which seems unlikely. Is there something I'm missing?
i don't know, but if i understand http://3taps-statistics.qatro.com/craigslist/index.pl correctly then they are missing lots of posts. the numbers seem to be percentages.
http://3taps.com/stats
Scroll down to the "daily report" and they've dropped from almost 2 million posts a day to 153.
Re: 3taps says CL blocking "all general search engines"
#10Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the last hour.
They probably did it through the webmaster tools portals provided by the search engines themselves. 3Taps has gone from indexing ~2 million posts a day to virtually 0. I did a couple site:cityname.craigslist.org searches on Google, restricted to the past 24 hours, and got no results.