Poll: Should this site be visible to Yahoo, MSN crawlers? • Hacker News Board

msg wrote: post #12

noelchurchill wrote: post #8

why not block all search engines?

http://www.google.com/search?q=site%3Anews.ycombinator.com+y... There's no other search.

> There's no other search.

http://www.searchyc.com

(edit: I'm not the one who voted you down)

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#15

Post by jsrn » Wed, Apr 16, 2008, 5:54 PM UTC

kirubakaran wrote: post #14

msg wrote: post #12

Earlier quoted context omitted.

http://www.google.com/search?q=site%3Anews.ycombinator.com+y... There's no other search.

> There's no other search. http://www.searchyc.com (edit: I'm not the one who voted you down)

and

http://nycs.bigheadlabs.com/

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#16

Post by agentbleu » Wed, Apr 16, 2008, 5:57 PM UTC

google knows how to crawl, slurp 3 is out just today so maybe yahoo will improve but even if you give it to them on a plate (sitemap.xml) they still seem to screw it up. The traffic is not worth their bandwidth imo.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#17

Post by icey » Wed, Apr 16, 2008, 5:57 PM UTC

Why not just robots.txt out all search engines? Of course, if you did that, it'd be pretty nice to have a "Search" button up there in the orange bar.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#18

Post by pchristensen » Wed, Apr 16, 2008, 5:59 PM UTC

jules wrote: post #9

Why does it matter how many useless urls they visit? If you're trying to fix the "expired link" issue -- there are better ways to fix that. I think you should ban google too if you ban the others. The only disadvantage I can see is that people from here who want to find something they've seen on news.yc can't use google anymore.

Blocking a crawler is a one liner that could be done in a couple seconds. Tackling the limitations of a continuation based architecture is a bigger problem.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#19

Post by pchristensen » Wed, Apr 16, 2008, 6:01 PM UTC

jsjenkins168 wrote: post #5

My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.

Not evil at all. Letting someone crawl your site is a business decision. If you think the resource cost on your servers is worth the added traffic that the search engine can send, then go for it. For this site, I would guess the resource cost is more (because of storing continuations) and the benefit of traffic is less (since pg doesn't even want it). Therefore, it's a good decision on pg's part and Yahoo's loss.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#20

Post by pchristensen » Wed, Apr 16, 2008, 6:01 PM UTC

amichail wrote: post #2

Rather than looking for ways to constrain growth, why not use this as an opportunity to invent a better social news site?

I think constraining growth IS the way he's trying to invent a better news site.