Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Suddenly, Hacker News is not the first result for 'Hacker News'

google.com

Post reply on HN
Print view View original

11–20 of 216 posts

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#11

Post by markerdmann »

Has Google been making significant changes to the search ranking algorithms in the past couple months? I've noticed a significant decline in the quality of results, to the point that (for the first time in years), I've bounced over to DuckDuckGo or Bing to try my luck there. I love Google as a company, so (if this isn't just in my head) I'd love to see things get better again.

EDIT: Looks like the change to HN's ranking is related to a change that pg made, so my comment is now less relevant to the parent post. I still stand by it, though. :-)

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#12

Post by [deleted] »

Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

[deleted]

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#13

Post by moe »

Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

Does it perhaps return something to the tune of "Unknown or expired link"? :-P

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#15

Post by pg »

Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

I sent you an email about this.

(A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#16

Post by RBerenguel »

Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

Advantages of having a deep Google guy as a user :)

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#17

Post by Matt_Cutts »

moe wrote: post #13
Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

Does it perhaps return something to the tune of "Unknown or expired link"? :-P

Nope, it looks like we haven't been able to fetch even the robots.txt file for a few days, so our system is trying to play it safe by assuming that HN is unreachable/dead.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#18

Post by tibbon »

pg wrote: post #15
Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling?

Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#19

Post by pierrefar »

pg wrote: post #15
Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Hi Paul,

A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling.

If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site.

Let me know if you need more help.

Edit Detailed instructions to set a custom crawl rate:

1. Verify the site in Webmaster Tools.

2. On the site's dashboard, the left hand side menu has an entry called Site Settings. Expand that and choose the Settings submenu.

3. The page there has a crawl rate setting (last one). It defaults to " Let Google determine my crawl rate (recommended)". Select "Set custom crawl rate" instead.

4. That opens up a form and choose his desired crawl rate in crawls per second.

If there is a specific problem with Googlebot, you can reach the team as follows:

1. To the right hand side of the Crawl Rate setting is a link called "Learn More". Click that to open a yellow box. 2. In the box is a link called Report a problem with Googlebot which will take you to form you can fill out with full details.

Thanks!

Pierre

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#20

Post by Matt_Cutts »

pg wrote: post #15
Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.
Post reply on HN