Live data from Hacker News

Poll: Should this site be visible to Yahoo, MSN crawlers?

news.ycombinator.com

41–50 of 98 posts

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#41
post #38

Allow all or ban all. let's make fairness a rule on the internet, it gives a chance to all startup to emerge by giants side. If Yahoo was not fair to Google no one knows what would have happened.

That statement is roughly equivalent to "imprison no one or imprison everyone".

Isn't that fair?

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#42
post #39
post #12

Earlier quoted context omitted.

http://www.google.com/search?q=site%3Anews.ycombinator.com+y... There's no other search.

Does that actually work for people? I've tried using Google to search News.YC for an older topic on a couple of occasions, and the results have been very hit-or-miss. Is it just me?

(plug)

If you remember some key words from the title, you can do a quick find in http://www.kirubakaran.com/phr0zen (snapshots of front page). It is not really search search but I find it useful all the time.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#43

why not block all search engines?

Because News.YC doesn't have it's own search engine. So I use google to search the site.

Why not solve the real problem and set up a search engine for News.YC? This has other advantages such as controlling what shows up in the results, different ways to sort them, etc.

It's easy to do with open-source software such as Solr (lucene.apache.org/solr) or Hounder (hounder.org, shameless plug because it is our product), which powers the wordpress.com site.

Edit: it looks like someone already did this: searchyc.com.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#44
I would set NOFOLLOW but not bother with NOINDEX, assuming those crawlers handle the same meta tags google does. That way news.yc pages people link to from other sites would be indexed, but x urls would be left alone.

I do not currently use Yahoo or MSN search, but would rather not have the option of using them cut off.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#47
post #33
post #27

I agree to the site being invisible to yahoo, msn, ask, etc. At the same time, please let the newer/upcoming search engines crawl the site, else, google would become IE circa 2000-2004 and there would be very little innovation in search. my 2 cents.

This patchwork deny/allow is nearly as bad as banning everyone but Google. Every small site will make the decision of who's young enough to deserve a chance differently -- making only Google the safe choice for searching. A principle of equal access by any well-behaved search engine is the only policy which supports startups and diversity. The market already has a winner-take-all quality; why make it even worse for c…

I don't mean it to be a patchwork system, I should clarify, a simple deny Yahoo, MSN works fine with me since Paul is making a business decision. What is not fine with me is allowing only google, and denying everyone else.

Like you, I support startups and want diversity and innovation, which is why I would like search startups to crawl this website.

Hope that clears it up.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#48

why not block all search engines?

Googling for my name lists my Hacker News profile as one of the top results. I think that is really cool.

That is cool. My username returns my profile here as the number one hit. I had no idea it would score so highly.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#49
post #46

Earlier quoted context omitted.

Isn't that fair?

That is also hopelessly naïve.

You don't have to be insulting to make a point. I understand you may be using Google and like it, you must think about people using Yahoo as well. It is selfish of yours to base your judgment on your preference. If I'm missing something let me know.
Post reply on HN