Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Suddenly, Hacker News is not the first result for 'Hacker News'

google.com

Post reply on HN
Print view View original

21–30 of 216 posts

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#21

Post by Matt_Cutts »

tibbon wrote: post #18
pg wrote: post #15

Earlier quoted context omitted.

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling? Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.

We do have a self-service tool in webmaster tools for people who prefer to be crawled slower.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#22

Post by w1ntermute »

Out of curiosity, how did the OP even find out that this was going on? He appears to be a long-time user of the site, and therefore would have no reason to Google "Hacker News".

The most common other reason would be that some people use Google as their URL bar - instead of typing "hackerne.ws" or "news.ycombinator.com" into the URL bar, they type "Hacker News" into Google and click on the first result. However, I would've thought that the types of people using HN would have the tech savvy to use a keyword bookmark, or at least the URL.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#23

Post by ashishgandhi »

Matt_Cutts wrote: post #20
pg wrote: post #15

Earlier quoted context omitted.

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.

Does that mean at Google you can manually set the system to treat different sites differently?

I don't mean the ranking but other aspects - like you guys blacklisted some domains which produce low quality content in wholesale. (I don't know if the algorithm was tweaked to detect and filter such sources or if it was a manual thing.)

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#24

Post by Matt_Cutts »

markerdmann wrote: post #11

Has Google been making significant changes to the search ranking algorithms in the past couple months? I've noticed a significant decline in the quality of results, to the point that (for the first time in years), I've bounced over to DuckDuckGo or Bing to try my luck there. I love Google as a company, so (if this isn't just in my head) I'd love to see things get better again. EDIT: Looks like the change to HN's rank…

If you can look back in your search history to find the specific searches that didn't do well for you, we're always happy to get really concrete examples.

The best reports look like "I did a search for [Bavarian red widgets] and the results weren't good because e.g. you were missing a specific page X that you used to return or should return, or you returned Austrian red widgets" or whatever.

Lots of Googlers are clearly hanging out on HN over Thanksgiving while they're stuck at relatives' houses. :)

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#25

Post by aw3c2 »

pierrefar wrote: post #19
pg wrote: post #15

Earlier quoted context omitted.

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Hi Paul, A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling. If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site. Let me know if you need more help. Edit Detailed instructions to set a custom crawl rate: 1. Verify…

I would like to set that crawl rate but do not see why I must register at Google to do so. Why can't Google support the Crawl-Delay directive in robots.txt for this?

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#26

Post by narcissus »

w1ntermute wrote: post #22

Out of curiosity, how did the OP even find out that this was going on? He appears to be a long-time user of the site, and therefore would have no reason to Google "Hacker News". The most common other reason would be that some people use Google as their URL bar - instead of typing "hackerne.ws" or "news.ycombinator.com" into the URL bar, they type "Hacker News" into Google and click on the first result. However, I wou…

Could've been from telling a friend "hey yeah... found this on Hacker News". Then the friend went looking for 'Hacker News' on Google and came across the current top result.

At that point that friend asks the OP: "by Hacker News, you mean , right?". Then the OP goes looking...

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#28

Post by arctangent »

Matt_Cutts wrote: post #21
tibbon wrote: post #18

Earlier quoted context omitted.

Could you use some sort of sitemap or other way to provide the data to Google that isn't so damaging to site performance? Or in Google Webmaster tools turn down the rate of crawling? Just realized that this could be a problem for lots of sites, and I'm curious as to what the best solution is, since not everyone has Matt Cutts reading their site and helping out.

We do have a self-service tool in webmaster tools for people who prefer to be crawled slower.

Do you support/respect the "crawl-delay" directive?

http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#29

Post by hahla »

Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources.

Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news' to visit this site, but since when did that matter to google?

Also don't get me wrong I love both google and hackernews. I just find whats going on in this thread interesting..

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#30

Post by markerdmann »

Matt_Cutts wrote: post #24
markerdmann wrote: post #11

Has Google been making significant changes to the search ranking algorithms in the past couple months? I've noticed a significant decline in the quality of results, to the point that (for the first time in years), I've bounced over to DuckDuckGo or Bing to try my luck there. I love Google as a company, so (if this isn't just in my head) I'd love to see things get better again. EDIT: Looks like the change to HN's rank…

If you can look back in your search history to find the specific searches that didn't do well for you, we're always happy to get really concrete examples. The best reports look like "I did a search for [Bavarian red widgets] and the results weren't good because e.g. you were missing a specific page X that you used to return or should return, or you returned Austrian red widgets" or whatever. Lots of Googlers are clea…

That's helpful, thanks. I haven't kept a record of those searches (they were mostly programming-related queries), but I'd be happy to submit reports in the future. Is there a URL or email address for this purpose?
Post reply on HN