Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Suddenly, Hacker News is not the first result for 'Hacker News'

google.com

Post reply on HN
Print view View original

121–130 of 216 posts

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#121

Post by larrys »

billpatrianakos wrote: post #93
wmf wrote: post #89

Earlier quoted context omitted.

PG knows his shit. He knows how to block crawlers, he knows how get SEO done, he's no dummy. Apparently he doesn't use Google Webmaster Tools, though.

It seems he doesn't want to. From what I understand there's the possibility that he'd prefer to accomplish his goal without the use of Wemaster Tools. Like I said, he knows his shit, so he's going to do things on his terms if he can. Webmaster Tools might be what the average guy wants to use but then there are some who follow the beat of their own drummer whenever possible for whatever reasons we may never know.

Speaking of knowing ones shit your site listed in your profile billpatrianakos.com is dead - no dns being returned.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#122

Post by adbge »

Matt_Cutts wrote: post #106
officemonkey wrote: post #85

Earlier quoted context omitted.

I would think that the number of people who (a) know how to create a valid robot.txt file, (b) have some idea of how to use the "crawl-delay" directive and (c) write a "shoot-themselves-in-the-foot" worthy error is vanishingly small.

I alluded to some of the ways that I've seen people shoot themselves in the foot in a blog post a few years ago: http://www.mattcutts.com/blog/the-web-is-a-fuzz-test-patch-y... "You would not believe the sort of weird, random, ill-formed stuff that some people put up on the web: everything from tables nested to infinity and beyond, to web documents with a filetype of exe, to executables returned as text documents. In…

The number of webpages with HTML that's just plain wrong (and renders fine!) is staggering. I often wonder what the web would be like if web browsers threw an error upon encountering a syntax error rather than making a best effort to render.

If you're writing HTML, you should be validating it: http://validator.w3.org/

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#123

Post by einhverfr »

storborg wrote: post #72
arctangent wrote: post #28

Earlier quoted context omitted.

Do you support/respect the "crawl-delay" directive? http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl...

Nope. As mentioned above, apparently Google thinks people will "shoot themselves in the foot" with the crawl-delay directive, while they won't with Google's special interface (which requires registering and logging in).

What more can you expect from the World's Largest Adware Company?

Seriously, one thing about Google is that they seem to really like ensuring people are logged on, preferably at all times. Fortunately recent changes to Google Apps (promoting apps user accounts to full Google accounts) has made this more complex on my side and probably degraded the level of actionable info they can get out of it.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#124

Post by moses1400 »

hahla wrote: post #29

Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…

Completely agree with you Hahla - amazing to see the Googlers jump in including Matt Cutts so quickly to support PG and HN and YC yet 99.99999999% would never ever see support like this.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#126

Post by pg »

wheels wrote: post #113
pg wrote: post #15

Earlier quoted context omitted.

I sent you an email about this. (A couple weeks ago I banned all Google crawler IPs except one. Crawlers are disproportionately bad for HN's performance because HN is optimized to serve recent stuff, which is usually in memory.)

Have you considered putting a caching reverse proxy in front of the arc app to keep the backend from having to render all of the old pages? It seems like the only dynamic element of old articles is the "$x days ago" bit and that'd be pretty easy to turn into something static by instead just putting in timestamps in the actual HTML and using Javascript to transform them into how many hours / days ago they were. Then t…

I believe Rtm has already set one up.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#127

Post by einhverfr »

larrys wrote: post #120
billpatrianakos wrote: post #74

Earlier quoted context omitted.

I see your point and maybe my attitude toward it is totally biased or morally/ethically off balance but I think it's just fine. PG knows his shit. He knows how to block crawlers, he knows how get SEO done, he's no dummy. "So what?" you say. Well I work in web development and I meet with people all the time and explain to them how to run a site and get rankings up, etc. I tell them I'll give them the tools, show them…

PG: "A couple weeks ago I banned all Google crawler IPs except one" With regard to this issue doing the above is not evidence of knowing your shit in the area in question.

To be honest, I disagree. The fact is that people who know their shit shoot themselves in the foot occasionally. That's how they come by the shit they know. Timid people rarely know their shit because they are too afraid to make mistakes.

Being bold is an important aspect of getting to know your shit.

Also, besides this, this action has gotten us talking about how Google throttles crawling, and this discussion has further reduced my opinion of the World's Largest Adware Vendor.

Now if I were to ban Google searchbots based on what PG is saying I would simply limit them to the first page or two of listing. I think this can be done in a robots.txt?

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#128

Post by tillk »

ludwigvan wrote: post #51

A few months ago, I was talking about the Stanford ai/ml-classes to one of my friends. He asked me if there were any more classes and I said "Yes, take a look at the front page in Hacker News, there are some links. I visit that site frequently, it's very helpful." There was one little issue though. The poor guy didn't know what hackernews was, so found that site (hackernews.com). He then scanned the Twitter stream ov…

great story, bro

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#129

Post by Matt_Cutts »

jfoster wrote: post #88
Matt_Cutts wrote: post #9

I think I know what the problem is; we're detecting HN as a dead page. It's unclear whether this happened on the HN side or on Google's side, but I'm pinging the right people to ask whether we can get this fixed pretty quickly. Added: Looks like HN has been blocking Googlebot, so our automated systems started to think that HN was dead. I dropped an email to PG to ask what he'd like us to do.

Has Google considered an indicator next to results for this type of case? A single red exclamation mark with further details when the mouse is over it, for example. I'm aware of webmaster tools, but it seems not all webmasters are.

We do include some information in the search results for cases where our users could be harmed, e.g. if we detect malware or if we think that your site may have been hacked. We've been exploring other ways to alert site owners that there's something important affecting their site.

But people do choose to remove their sites, so we can't always tell between a mistake vs. someone who genuinely prefers not to be in Google's index.

Re: Suddenly, Hacker News is not the first result for 'Hacker News'

#130

Post by csallen »

hahla wrote: post #29

Does anyone else find it disturbing that google employees are bending over for pg/hn? Seriously, if any other webmaster blocked googles bots they wouldn't change their algorithms to accommodate, or see how they could use less of our resources. Its pg's fault not googles, and I dont see why they should care. Maybe from their standpoint it would be more beneficial to google users who are used to typing in 'hacker news'…

What's more interesting is that people ONLY feel the way you do when the company in question is big. If it's a tiny startup doing some sort of unscalable marketing or customer support, nobody blinks an eye. But if a Google-sized organization does it, it's shocking or disturbing.

I think your best bet is to realize that, regardless of size, companies are always made up of people. And people are opinionated, subjective, and prone to making decisions that fall outside of some standard set of rules.

Post reply on HN