Poll: Should this site be visible to Yahoo, MSN crawlers?

A large fraction of our http requests come from Yahoo crawlers-- almost 20% yesterday. Their crawler seems significantly stupider than Google's. Yesterday we got 12,423 requests from the Google crawler, of which 4148 were for x (= mostly useless) urls, and 43,087 requests from Yahoo crawlers, of which 30,652 were for x urls.

Unlike most sites, I'm looking for ways to constrain our growth. News.YC is deliberately not intended to become a massively popular site. So the thought occurred to me: why not just ban Yahoo crawlers? And MSN too, while we're at it. I don't know anyone who uses either of them for search. I'd just as soon have the site be invisible to them. But what does the community think? Do any hackers use Yahoo or MSN search?

maybe one way of constraining would be to limit access say by invitation only or some other method.

My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google.

As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.

Block 'em.

Why does it matter how many useless urls they visit? If you're trying to fix the "expired link" issue -- there are better ways to fix that.

I think you should ban google too if you ban the others. The only disadvantage I can see is that people from here who want to find something they've seen on news.yc can't use google anymore.