I actually think this might be positive for the quality of the conversation on this site.
True. No reason not to keep HN on the hush hush, eh?
Suddenly, Hacker News is not the first result for 'Hacker News'
41–50 of 216 posts
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#42I actually think this might be positive for the quality of the conversation on this site.
True. No reason not to keep HN on the hush hush, eh?
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#43Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…
From my personal point of view, I help webmasters when I see an issue. It's part of my job and it's something that gets me fired up. I help when I go on our forums, I do that here, and I'm regularly helping webmasters on Google+ (my team does regular webmaster hangouts that anyone can join - see my profile: https://plus.google.com/115984868678744352358 ) and also on Twitter. We can't cover everything, we can't be everywhere, but when we come across something we can help with, we try to help!
Another important point about this thread: this is a very common issue that regularly comes up, and no site is immune from it. I regularly see major sites have a firewall that auto-configures itself to block Googlebot, and the webmaster doesn't know what's going on. Raising awareness about this problem and how to fix it adds to the importance of replying.
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#44Earlier quoted context omitted.
I would like to set that crawl rate but do not see why I must register at Google to do so. Why can't Google support the Crawl-Delay directive in robots.txt for this?
My plane is about to take off, but very briefly: people sometimes shoot themselves in the foot and get it way, way wrong. Like "crawl a single page from my website every five years" wrong. Crawl-Delay is (in my opinion) not the best measure. We tend to talk about "hostload," which is the inverse: the number of simultaneous connections that are allowed.
Another great way to shoot yourself in the foot (and getting it way way wrong) is to block all Googlebot IPs except for one.
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#45Earlier quoted context omitted.
I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)
Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.
If only we could all get our Google woes fixed in such a manner.
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#46Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…
Was the query for a domain name or just hacker+news? How do we know?
Does make any difference?
I don't think of this site as "Hacker News". I think of it as ycombinator, and the subdomain, news.
Should users of hackernews.com think of that site as something else, e.g. whatever is between the title tags?
A searchable list of domain names, ranked by popularity. Or even a searchable list of main page titles. Is that how some users are using Google? If so, Google does not need a full, current cached copy of the crawlable web to provide that.
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#47I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.
Seeing threads like this remind me that HN is still a pretty tight-knit community of real people doing real things.
It's good to see this stuff sometimes. Thanks, Matt!
edit:
And then reading some of the other threads on this topic is a bit...something.
Guys, can you calm the conspiracy theory nonsense a bit? Please?
If you're not on this site very much, you might not realize that Matt pops into almost every thread where google is doing something strange regardless of who they're doing it to, and tries to help figure out what is happening. This isn't HN getting some sort of preferential treatment, this is just the effect of having a userbase full of hackers.
You'd see the same type of thing on /. years ago if you frequented it enough.
This is nothing new. This is what a good community looks like. Everybody relax.
Honestly if you read the things that Matt and Pierre have said, they just looked at "freshness" (I believe that is what it is called), and inferred that PG had blocked their crawlers.
This is all stuff you can get from within google webmaster tools (which isn't some secret whoooo insider google thing. It's something they offer to everybody, and it's just like analytics.)
OH! Wait! I mean (hold on, let me spin up my google conspiracy theory generator): thehackernews.com has more ads on it so google is intentionally tweaking their algo to serve that page at a higher point than the real HN because of ads!
DUH!
C'mon, guys, look at their user pages. They're both just active users of the site trying to help out.
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#48Earlier quoted context omitted.
Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling? Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.
We do have a self-service tool in webmaster tools for people who prefer to be crawled slower.
or faster : http://news.ycombinator.com/item?id=2382728
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#49[deleted]
Re: Suddenly, Hacker News is not the first result for 'Hacker News'
#50Out of curiosity, how did the OP even find out that this was going on? He appears to be a long-time user of the site, and therefore would have no reason to Google "Hacker News". The most common other reason would be that some people use Google as their URL bar - instead of typing "hackerne.ws" or "news.ycombinator.com" into the URL bar, they type "Hacker News" into Google and click on the first result. However, I wou…
When I'm on someone else's phone or computer, I just search for [hn] to geT here quickly.