why not block all search engines?
Because News.YC doesn't have it's own search engine. So I use google to search the site.
Did you not find searchyc.com useful?
Poll: Should this site be visible to Yahoo, MSN crawlers?
71–80 of 98 posts
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#72Earlier quoted context omitted.
Google uses Java, and since that is evidentally the only acceptable search engine, I'm not sure pg's posited elitism really goes that far.
I didn't say that PG would refuse to use any system that was developed in Java. What I meant that he would never would put himself in a situation where the application would depend on Java for development, or to add more features. To integrate with Google, you talk HTTP, not Java. To adopt Lucene, you'll have to talk Java. There is quite some difference, isn't it?
You can push everything you want indexed into SOLR by HTTP -- so even though it's Java/Lucene you don't have to get your hands dirty.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#73Why not just robots.txt out all search engines? Of course, if you did that, it'd be pretty nice to have a "Search" button up there in the orange bar.
How do you robot.txt for search engines? If anything, you could ban all robots, I suppose. I wouldn't like that - it is akin to those social networks where you can't get your data out.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#74I use Yahoo at home, from the Firefox toolbar. I'm growing increasingly suspicious of Google (call me crazy) and for many things it's about as good.
That said, I would have no problem with banning Yahoo from HN. I've never used Yahoo to search HN, and Yahoo's "site:foo" support is one of their relative weaknesses, so I doubt I ever will. (It "redirects to Site Explorer", which looks basically like normal search but uglier and with less information.)
You could even redirect them to The Site Which Shall Not Be Mentioned.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#75Earlier quoted context omitted.
You must be new here. The day PG & Co adopt some piece of software that is made in Java... well, I can't even finish the sentence.
Google uses Java, and since that is evidentally the only acceptable search engine, I'm not sure pg's posited elitism really goes that far.
They use some Java, but a whole lot more Python and C++...
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#76MSN's crawler is a spammer: they "check" using referral spam using the livebot IP addresses. They admitted to it, they said they stopped it, but they haven't.
Please ban them.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#77Earlier quoted context omitted.
Because News.YC doesn't have it's own search engine. So I use google to search the site.
Why not solve the real problem and set up a search engine for News.YC? This has other advantages such as controlling what shows up in the results, different ways to sort them, etc. It's easy to do with open-source software such as Solr (lucene.apache.org/solr) or Hounder (hounder.org, shameless plug because it is our product), which powers the wordpress.com site. Edit: it looks like someone already did this: searchyc…
Usualy Google gets better results that a custom search engine for your site, because Google also knows the links form other web pages to your site.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#78Earlier quoted context omitted.
Googling for my name lists my Hacker News profile as one of the top results. I think that is really cool.
That is cool. My username returns my profile here as the number one hit. I had no idea it would score so highly.
Not as cool as mine. My username returns a wikipedia page about some asteroid. Cool stuff.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#79Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#80lets block all search traffic and then take this even further -- lets have all incoming links get forwarded to a dummy hacker news with nothing but lame stuff so that anyone trying to access the site from a blog or something with an HTTP_REFERER will get redirected to the lame site. that'll definitely keep out the unwashed masses. :)
Even better: replace HN with a static snapshot of it as it exists right now! Then we know it will always be this good! :-D