Poll: Should this site be visible to Yahoo, MSN crawlers?

> [...] can't use google anymore.

I think pg should just make a search link on top of the site pointing to

http://www.searchyc.com/

or

http://nycs.bigheadlabs.com/

I can understand not caring how many people read the site, so long as the hacker community is well served. I guess I'm just not quite clear on the downside of growth outside of this community; more useless comments? Comment spam? I also wonder if blocking those Yahoo and MSN crawlers would make it harder for them to compete with Google, because I consider competition a good thing. Eventually, by allowing them access, they may continue to improve their search product.

Perhaps most importantly, I worry that blocking these crawlers might be seen as a political statement against Yahoo and MSN, one which they might take into account when considering the acquisition of a ycombinator funded startup.

-- Fred

Either ban them all (Google included) or ban none. But if you must, just limit what they can and cannot crawl.

I use Yahoo search occasionally and have often been impressed at it finding exact material, with a precise search, that Google couldn't.

Separate from the local implications on News.YC, society benefits by having multiple competing search engines. Seceding from all search indexes but Google contributes to a monopoly/monoculture.

Can you just warn off the stupid crawlers from valueless URLs with some combination of robots.txt, META robots directives, and 'nofollow'? (I'm happy to help craft these signposts, with a description of the URLs to be spared.)