Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Suddenly, Hacker News is not the first result for 'Hacker News'

google.com

Post reply on HN
Print view View original

31–40 of 216 posts

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#31

Post by joelhaasnoot »

pierrefar wrote: post #10

Hi I work at Google helping webmasters. It seems something has been blocking Googlebot from crawling HN, and so our algorithms think the site is dead. A very common cause is a firewall. I realize that pg has been cracking down on crawlers recently. Maybe there was an unexpected configuration change? If Googlebot is crawling too fast, you can slow it down in Webmaster Tools. I'm happy to answer any questions. This is…

Why is it that "our algorithms think the site is dead" so soon, yet when I search for a keyword, I still get sites that our dead or no longer contain the given keyword? Bad algorithms or is this "thinking dead" a time delayed thing?

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#32

Post by Matt_Cutts »

joelhaasnoot wrote: post #31
pierrefar wrote: post #10

Hi I work at Google helping webmasters. It seems something has been blocking Googlebot from crawling HN, and so our algorithms think the site is dead. A very common cause is a firewall. I realize that pg has been cracking down on crawlers recently. Maybe there was an unexpected configuration change? If Googlebot is crawling too fast, you can slow it down in Webmaster Tools. I'm happy to answer any questions. This is…

Why is it that "our algorithms think the site is dead" so soon, yet when I search for a keyword, I still get sites that our dead or no longer contain the given keyword? Bad algorithms or is this "thinking dead" a time delayed thing?

Normally there's a lag between when a site goes dead and when we crawl the site to see that it's dead.

It's also tricky because you don't want a single transient glitch to cause a site to be removed from Google, so normally our systems try to give a little wiggle room in case individual sites are just under unusual load or the site is down only temporarily.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#33

Post by unreal37 »

hahla wrote: post #29

Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…

It's not disturbing at all. I am glad Google is a company made up of humans and not uncaring robots. Human nature is that people want to help others when they are in a position to do so. I would do the same if HN needed my help.

Of course, there are people who want to take advantage of people like Matt to give themselves an economic advantage, and I have no sympathy for those people if they can't get someone in Google to help them as easily.

It comes down to karma.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#34

Post by Matt_Cutts »

aw3c2 wrote: post #25
pierrefar wrote: post #19

Earlier quoted context omitted.

Hi Paul, A site can be crawled from any number of Googlebot IP addresses, and so blocking all except one doesn't help in throttling crawling. If you verify the site in Webmaster Tools, we have a tool you can use to set a slower crawl rate for Googlebot, regardless of which specific IP address ends up crawling the site. Let me know if you need more help. Edit Detailed instructions to set a custom crawl rate: 1. Verify…

I would like to set that crawl rate but do not see why I must register at Google to do so. Why can't Google support the Crawl-Delay directive in robots.txt for this?

My plane is about to take off, but very briefly: people sometimes shoot themselves in the foot and get it way, way wrong. Like "crawl a single page from my website every five years" wrong.

Crawl-Delay is (in my opinion) not the best measure. We tend to talk about "hostload," which is the inverse: the number of simultaneous connections that are allowed.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#36

Post by zerostar07 »

hahla wrote: post #29

Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…

Disturbing yes but not surprising, HN is influential. Now whether it's hacking news or entrepreneur news is another issue.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#37

Post by Kiall »

ashishgandhi wrote: post #23
Matt_Cutts wrote: post #20

Earlier quoted context omitted.

Gotcha--thanks, Paul. I'm about to get on a plane, but we'll get this figured out where we're not sending as much hostload toward HN.

Does that mean at Google you can manually set the system to treat different sites differently? I don't mean the ranking but other aspects - like you guys blacklisted some domains which produce low quality content in wholesale. (I don't know if the algorithm was tweaked to detect and filter such sources or if it was a manual thing.)

No, It sounds like @pg has blocked google bot via a firewall, So as far as google's automated systems are concerned, hacker news was shutdown and thus removed from the index.

I believe Matt is simply offering help/advice on figuring out the best solution for HN.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#38

Post by StatHacking »

This is an example of why Goog's search algorithms (and others') should be open: http://news.ycombinator.com/item?id=3268371

A subtle attack may be by making bots stop indexing it or using SEO practices to lower it enough so it would become unsearchable, and therefore, non-existent.

Or just crack into Google...

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#39

Post by Kiall »

hahla wrote: post #29

Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…

I wouldn't call offering help/advice to a site you use as "bending over", I reckon any Google employee with access to the right tools would look into why one of their favourite sites has disappeared from the index..

Oh and, It could also have been a sign of a pretty major bug, considering the popularity of HN ;)

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#40

Post by AznHisoka »

hahla wrote: post #29

Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…

Yes, it's VERY disturbing, no question about it. Matt can sound all helpful, and cuddly, and oxytocin-inducing all he wants, but he's basically stepping in and helping 1 site over another, when the whole process should NOT favor anyone. There are lots of webmasters out there that do stupid stuff like override a robots.txt accidentally, yet I don't see Matt sending them an email, asking them to check up on it.. Come'n... this is lame!
Post reply on HN