Viewing profile — sigmadelta

But to match user queries, you need to have scraped the craigslist content, right? Isn't that against their TOU?

craigslist didn't change robots.txt: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT

http://blog.sfgate.com/techchron/2012/08/10/craigslist-backs... "One data harvester, 3taps, said earlier this week that Craigslist had blocked search engines such as Google from in…

His 3Taps service started as an exchange for classifieds, which he said would be pulled from eBay, craigslist, indeed, and others. But at some point all of the other vendors got dr…

Saying that "newyork.craigslist.org does not have any robots.txt and can be crawled as you like" is false. Search engines follow redirects until valid robots.txt files are found. F…

http://www.seroundtable.com/archives/019533.html That article is from 2009, noting that CL had added the meta tag to posting pages. So I think this is old news. I think storborg ha…

https://twitter.com/markmilian/statuses/233015694432813057 Mark Milian ‏@markmilian 7 Aug Contradicting earlier statement, 3Taps spokeswoman emails to say, "Craigslist is still all…

Just tried again, and I got a bunch of results: http://www.google.com/search?q=site%3Asfbay.craigslist.org+b... Of the 10 results on the first page, 7 are within the last hour. (1)…

Those tags aren't as obvious in interpretation as they first appear. A full write-up is a http://noarchive.net/meta/ for completeness, but in short: NOARCHIVE: ok to index this pag…

http://www.sfgate.com/technology/businessinsider/article/Cra... Not sure I agree with most the conclusions drawn in that article. The article does say that "sure enough, Google dis…

http://news.ycombinator.com/item?id=4348808

http://news.ycombinator.com/item?id=4348808

http://news.ycombinator.com/item?id=4348808

Did these people do any fact checking? I just ran another test: visit http://sfbay.craigslist.org/bia/ spot ad for a kids Giant MTX 250 bike http://sfbay.craigslist.org/nby/bik/318…

I just tried the following: visited http://sfbay.craigslist.org/bia/ saw ad for "Burning Man Beach Cruiser Bicycle" http://sfbay.craigslist.org/eby/bik/3189652975.html performed th…

http://www.theverge.com/2012/8/7/3225476/craigslist-blocks-3...

If that's true, how does 3taps know that CL is blocking search engines? Also: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified:…

Actually the part about search engines doesn't seem to be true... I just performed searches using Google, Yahoo, and Bing and got links to CL postings that were made within the las…

homepage pop-up: "At approximately noon on Sunday August 5th, Craigslist instructed all general search engines to stop indexing CL postings -- effectively blocking 3taps and other …