Earlier quoted context omitted.
> There's no other search. http://www.searchyc.com (edit: I'm not the one who voted you down)
and http://nycs.bigheadlabs.com/
They stopped indexing looong ago.
Poll: Should this site be visible to Yahoo, MSN crawlers?
61–70 of 98 posts
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#62Earlier quoted context omitted.
You don't have to be insulting to make a point. I understand you may be using Google and like it, you must think about people using Yahoo as well. It is selfish of yours to base your judgment on your preference. If I'm missing something let me know.
I sincerely apologize. I deserved that. My grandfather passed away today.
Sorry for your loss.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#63for someone (like me) using Yahoo's web search api (which is far better than Google's imho), it would be ideal if you left Yahoo's crawler to do it's job here.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#64My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.
Okay, that's one reason for not indexing the site. However, the poll question implies blocking it for traffic reasons. I'd vote for robots.txt'ing out the /x URLs, then re-running the vote with the question of "should we deliberately ban search engines so that we get less growth in visitors."
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#65I'm feeling kinda dumb here, but what are x urls?
They're short-lived URLs on this site like the one I'm using to add this reply:
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#66My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.
If we're sacrificing equal competitive access just to "constrain growth as much as possible", maybe we could block certain browsers, too? IE users aren't smart enough to change their defaults; Safari users are mindless Apple fanboys; Opera users are weird and talk with funny accents. Smart hackers will take the trouble to launch FF to access News.YC. Anything to preserve the quality of content here!
No, FF users are overzealous advocates of FOSS in general and Firefox in particular. Links, Lynx, and w3m users are too proud of their terminals. Netscape users don't realize it's the 21st century, AOL users are idiots. Flock and Seamonkey users are weird. No comment on Konqueror and Epiphany.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#67Earlier quoted context omitted.
Why not solve the real problem and set up a search engine for News.YC? This has other advantages such as controlling what shows up in the results, different ways to sort them, etc. It's easy to do with open-source software such as Solr (lucene.apache.org/solr) or Hounder (hounder.org, shameless plug because it is our product), which powers the wordpress.com site. Edit: it looks like someone already did this: searchyc…
You must be new here. The day PG & Co adopt some piece of software that is made in Java... well, I can't even finish the sentence.
Google uses Java, and since that is evidentally the only acceptable search engine, I'm not sure pg's posited elitism really goes that far.
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#68I use both Yahoo and Google.
Why can't you just use or a robots file to stop them from indexing x-links?
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#69Earlier quoted context omitted.
You must be new here. The day PG & Co adopt some piece of software that is made in Java... well, I can't even finish the sentence.
Google uses Java, and since that is evidentally the only acceptable search engine, I'm not sure pg's posited elitism really goes that far.
I didn't say that PG would refuse to use any system that was developed in Java. What I meant that he would never would put himself in a situation where the application would depend on Java for development, or to add more features.
To integrate with Google, you talk HTTP, not Java. To adopt Lucene, you'll have to talk Java. There is quite some difference, isn't it?
Re: Poll: Should this site be visible to Yahoo, MSN crawlers?
#70Is there a way to block only the useless calls? The x that fail? Or to robots.txt anything that isn't main story pages so it wouldn't follow every individual comment link?
It might be good to still be indexed on the primary single page for each discussion... Then again I only use google, so it doesn't bother me any.