Ask HN: Whoa - why did Google bork us?

Google is suddenly linking to a test site, not our main site. Does PageRank not matter?

Looks like Google borked us, and with it, a lot of the effort we have done is currently being flushed down the drain.

Searching for our more common keywords (amahi, linux home server, linux home networking, linux server, etc.) leads to Google displaying (at least to many people), not our long-established, PageRank-6, award-winning, super-duper, url (http://www.amahi.org) but our internal site called admin:

   admin dot amahi dot org

as the link for Amahi!?! (forgive me that I not to link to it here!)

It doesn't even show the sitelinks they used to show because the admin site does not have them, according to Google's Webmaster tools (WMT). There are no links on the web for our admin site (we only use it internally), near as I (or WMT) can tell.

Also, the description is not the metadata description in our site, but it's picked up from a

somewhere down in the first page of the site. (I just noticed Bing also having this same description so, unless they stole it from Google, maybe it's our fault.)

We went to Google WMT and claimed the admin site. The content was identical to the main site most of the time. It allowed us to test new things in it and it was meant to be private! As of a few hours, it's doing nothing but 301s to www.amahi.org bug Googlebot is still slurping it.

Any ideas what went wrong here?

  - How can this happen?
    - How did their crawler find the admin site??
      - referrers?
      - a toolbar? (we don't use any - though maybe once or twice we did for quick testing)
      - ????
    - Does PageRank and their famous incoming link equations not matter?? How can this be so off?
    - Have we been punished? If so, for what? (we're not doing anything even remotely fishy - it's a nice burgeoning community)
  - How long will this last?
    - We have gone to WMT to dramatically accelerate the crawl rate of www.amahi.org
    - We have set the admin site to 301 all the way into the main site
    - We have asked WMT to not remove the admin site from the search index
  - More importantly, how can we prevent this from happening again?
    - More proactive monitoring?
    - Any recommended ways to do this efficiently?

Our traffic and users has been trending down for a few days now and we're not sure how long has been going on (we just started to look at the logs and Googlebot has been at it for a little while).

We had never seen this before. I understand their crawler has to be very aggressive, but it seems to me like the indexing is broken, at least in this case.

Imagine if someone gets all our pages and puts them somewhere else. Even without links to them, they could take over us.

Thanks for the idea of the robots.txt. Since the admin site is really for testing and eventually released, it's some work to make the robots.txt dynamic. May do it soon just to make sure.

Having an admin or test site publicly accessible is a really bad idea, for exactly this reason. If any search engine comes across a link to this site, it will crawl the site. Even if it doesn't promote the test site above you main site, it can cause a lot of problems: * googlebot finds a ton of new content, and starts crawling this new content rather quickly, increasing load on your machines * since multiple domains are detected serving the same content, it could make your site look spammy and decrease your PR * there's no guarantee that your main site content will show up first in google search results, which is obviously bad

I worked at a site that encountered this problem in a similar manner. The way it worked was the main site was www.example.com, and we had a CDN + squid in front of static.example.com. Only static assets like images were served from the CDN version of the site. However, the code didn't actually enforce this (i.e. it was purely enforced by only linking to static assets). Somehow Google found a link to the cached copy of the site and started crawling everything. Badness ensued.

It sounds like your situation is pretty easy to fix -- enforce that only internal IPs can access the test site (or require a special cookie, etc.). In the previous example that I gave, we had to explicitly whitelist certain servlets as acceptable to be served from the static site (and enforce that in the code).

Well, we found the issue!

I started to look at the WMT crawl info and there were a bunch of network unreachable errors.

Here is what happened: our site being PR6/7, we started getting a lot of scans for exploits. Some of them over and over. A few months ago I wrote a rule-based procedure to automatically block these IPs at the firewall level.

Well, someone had tricked Googlebot into searching our site by using a "break in" URL, so Googlebot fell trap to that and got blocked.

Interestingly, most of the access to our site from Google were from one IP and it was blocked for a little while.

In the end, since the "admin" site was a copy of the main site, it effectively acted as the backup!

We've taken steps in the rules to not block Googlebot :)