Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Poll: Should this site be visible to Yahoo, MSN crawlers?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#81

Post by noodle »

icky wrote: post #80
noodle wrote: post #40

lets block all search traffic and then take this even further -- lets have all incoming links get forwarded to a dummy hacker news with nothing but lame stuff so that anyone trying to access the site from a blog or something with an HTTP_REFERER will get redirected to the lame site. that'll definitely keep out the unwashed masses. :)

Even better: replace HN with a static snapshot of it as it exists right now! Then we know it will always be this good! :-D

i was thinking that it would be funny if it was a data scrape of digg with a HN skin applied.

although that would probably turn out badly

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#82

Post by yters »

jsjenkins168 wrote: post #5

My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.

It'd be interesting if source browsing and javascript hacking were the only ways to affect the site. Then only hackers would be able to contribute.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#83

Post by yters »

superkarn wrote: post #36

I voted to remain visible. I assume that resource is not a problem for YC. This fact, coupled with YC not having a search feature, leads me to believe that it's better for YC community that this site is indexed by more than just one search engine. One big reason, as pointed out by Tichy, is that not all of us use Google as our primary search engine. By cutting off Yahoo, MSN, etc, but not Google, you would essentiall…

What's the slippery slope?

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#84

Post by derefr »

gojomo wrote: post #51
jsjenkins168 wrote: post #5

My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.

If we're sacrificing equal competitive access just to "constrain growth as much as possible", maybe we could block certain browsers, too? IE users aren't smart enough to change their defaults; Safari users are mindless Apple fanboys; Opera users are weird and talk with funny accents. Smart hackers will take the trouble to launch FF to access News.YC. Anything to preserve the quality of content here!

If you want to really constrain it, require a browser specific to YC... perhaps written in Arc. Also: secret handshake ;)

(I can think of a justification for the first part, though--the time it takes to launch a separate browser that one cannot use for any other sites is the perhaps near the time it takes to cool down and forget any external emotional influences on your reading.)

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#88

Post by ntoshev »

I think it is a horrible idea.

Blocking search engines is evil. This site helps the search engines(allowing them to rank other results better) and the search engines helps people (by providing the missing search functionality and allowing users to find useful content, on this site or elsewhere).

Blocking traffic from MSN and Yahoo might bring less non-hackers to the site, but it would also introduce bias in the community that is not good. I don't want Microsoft hackers (presumably using MSN search) to be excluded from this community.

I believe nowadays Google is only marginally better than Yahoo or Msn search. I don't need more hacker fashions.

The last thing is, the problem with the crawlers and traffic load is easily solvable using robots.txt. Don't mess with the system more than you need.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#89

Post by ntoshev »

yters wrote: post #83
superkarn wrote: post #36

I voted to remain visible. I assume that resource is not a problem for YC. This fact, coupled with YC not having a search feature, leads me to believe that it's better for YC community that this site is indexed by more than just one search engine. One big reason, as pointed out by Tichy, is that not all of us use Google as our primary search engine. By cutting off Yahoo, MSN, etc, but not Google, you would essentiall…

What's the slippery slope?

At least two dimensions:

- tying the site to specific products. pg may decide firefox browser is the best predictor for being a hacker and ban all other browsers. Would you like that?

- introducing bias to the community. the search engine may be a correct predictor for being a hacker, but what about the false negtives: do we want to exclude the hackers working on MSN search engine from this community, for example? I certainly don't.

Re: Poll: Should this site be visible to Yahoo, MSN crawlers?

#90

Post by ntoshev »

pchristensen wrote: post #19
jsjenkins168 wrote: post #5

My vote is to constrain growth as much as possible, at least that which comes from stupid sources. Smart hackers will find this site just fine without Yahoo or MSN, probably even Google. As "evil" as blocking sites and crawlers may sound, I think these types of measures will be necessary to preserve the quality of content here. Whatever actions further that objective have my vote.

Not evil at all. Letting someone crawl your site is a business decision. If you think the resource cost on your servers is worth the added traffic that the search engine can send, then go for it. For this site, I would guess the resource cost is more (because of storing continuations) and the benefit of traffic is less (since pg doesn't even want it). Therefore, it's a good decision on pg's part and Yahoo's loss.

It is evil. A lot of the evil things out there are business decision (e.g. Microsoft monopoly practices). Even if this decision has a positive net effect on the community, as a result the world is likely to be worse off (all non-google searchers get suboptimal search, on this site and elsewhere).
Post reply on HN