Ask HN: What do you do when your website seems to be penalised by Google?

#1

Post by mmavnn » Mon, Jan 12, 2015, 10:27 AM UTC

I have a personal blog (mostly dev related); it's been going for a while. On a couple of specialist subjects (F# type providers being the main example) some of the posts are reasonably popular and linked to by many other people. Although it's a small site, on these subjects it tends to show up in the first page on Bing, Duck Duck Go, etc for searches like "Type provider tutorial" and right at the top if you use a specific phrase (like the title of my most popular post, "Type Providers from the Ground Up".

Google hates it. Basically, however specific the query, my blog never turns up unless you actually put the base url into your query. Ironically, plenty of spam sites' copies of the posts appear quite high in the search results.

What do you do in these types of situations? I've done no SEO beyond writing content, so I'm pretty sure I've used no "black hat" techniques. I've no ads, no duplicate content. Google webmaster tools claims the site is not blacklisted and that there is nothing wrong with it.

It feels wrong and possibly pointless to start again several years down the line with a new url just because Google doesn't seem to like the current one; but on the other hand, the lack of organic search results will always be a limit on the readership. For a personal blog this is irritating and disappointing - if I was freelance or this was my company blog, it would be a real and immediate financial hit.

Thoughts or advice for people facing this situation?

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#2

Post by rfergie » Mon, Jan 12, 2015, 10:53 AM UTC

Have you registered your site with Google webmaster tools (https://www.google.com/webmasters/tools/)?

That would be a good first step for seeing if Google are having specific trouble with anything on your site

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#3

Post by patio11 » Mon, Jan 12, 2015, 10:55 AM UTC

This is a hard thing to debug from outside the Googleplex, but you are currently serving a canonical tag:

" rel="nofollow">http://blog.mavnn.co.uk/type-providers-from-the-ground-up">

for a URL which cannot possibly return an HTTP 200. (It 301s to a URL with a / on the end.)

This combination could cause Google to conclude that you have no page which requires inclusion in their main index.

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#4

Post by mmavnn » Mon, Jan 12, 2015, 11:00 AM UTC

rfergie wrote: post #2

Have you registered your site with Google webmaster tools ( https://www.google.com/webmasters/tools/ )? That would be a good first step for seeing if Google are having specific trouble with anything on your site

I have yes, no mention of any issues from the tools (it even claims to like the sitemap).

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#5

Post by mmavnn » Mon, Jan 12, 2015, 11:03 AM UTC

patio11 wrote: post #3

This is a hard thing to debug from outside the Googleplex, but you are currently serving a canonical tag: " rel="nofollow">http://blog.mavnn.co.uk/type-providers-from-the-ground-up"> for a URL which cannot possibly return an HTTP 200. (It 301s to a URL with a / on the end.) This combination could cause Google to conclude that you have no page which requires inclusion in their main index.

Interesting; that's autogenerated by Octopress (or Jakyll). I'd never noticed it's missing the '/'.

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#6

Post by ovi256 » Mon, Jan 12, 2015, 11:04 AM UTC

patio11 wrote: post #3

This is a hard thing to debug from outside the Googleplex, but you are currently serving a canonical tag: " rel="nofollow">http://blog.mavnn.co.uk/type-providers-from-the-ground-up"> for a URL which cannot possibly return an HTTP 200. (It 301s to a URL with a / on the end.) This combination could cause Google to conclude that you have no page which requires inclusion in their main index.

Is it irony or something else that a system of supposed vast computing power and learning (and certain real world power via its distribution of search riches) is broken by a tiny thing like this fix for a missing slash ?

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#7

Post by kevinbowman » Mon, Jan 12, 2015, 11:17 AM UTC

Presumably all of the spam copies are actually harming your rankings as well, as Google will see those as duplicate content.

(Note that I'm guessing here, I have no particular authority in the area)

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#8

Post by mmavnn » Mon, Jan 12, 2015, 11:23 AM UTC

mmavnn wrote: post #5

patio11 wrote: post #3

This is a hard thing to debug from outside the Googleplex, but you are currently serving a canonical tag: " rel="nofollow">http://blog.mavnn.co.uk/type-providers-from-the-ground-up"> for a URL which cannot possibly return an HTTP 200. (It 301s to a URL with a / on the end.) This combination could cause Google to conclude that you have no page which requires inclusion in their main index.

Interesting; that's autogenerated by Octopress (or Jakyll). I'd never noticed it's missing the '/'.

Known Octopress issue, apparently: http://hackingoff.com/blog/octopress-default-seo-flaws/

Thanks, patio11

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#9

Post by JVerstry » Mon, Jan 12, 2015, 11:43 AM UTC

First, make sure your sitemap.xml is exhaustive. Then, check the number of indexed pages in Google Webmaster Tool (after a couple of days if you had to update you sitemap). If few pages are indexed, go through this checklist at https://ligatures.net/content/expertise/site-not-indexing-ch... to fix possible issues. If your pages are still not displayed in search results, then you are likely another victim of a well-known chicken-and-egg problem for content based sites: you need links for ranking and you need ranking to attract links. Yet, most niche are saturated and you are likely crushed by competition. The only efficient way out is to obtain dofollow backlinks from sites/blogs which are: i) Not under your control (i.e., a forum profile link is under your control...) ii) Editorially reviewed iii) Have relevant topics to yours iv) Which are already trusted by Google v) Which are popular Other links won't make much of a difference.

Re: Ask HN: What do you do when your website seems to be penalised by Google?

#10

Post by thaumaturgy » Mon, Jan 12, 2015, 11:43 AM UTC

ovi256 wrote: post #6

patio11 wrote: post #3

This is a hard thing to debug from outside the Googleplex, but you are currently serving a canonical tag: " rel="nofollow">http://blog.mavnn.co.uk/type-providers-from-the-ground-up"> for a URL which cannot possibly return an HTTP 200. (It 301s to a URL with a / on the end.) This combination could cause Google to conclude that you have no page which requires inclusion in their main index.

Is it irony or something else that a system of supposed vast computing power and learning (and certain real world power via its distribution of search riches) is broken by a tiny thing like this fix for a missing slash ?

URLs are really, really, really, really hard to get right on a large scale. For a side project I've written my own crawler/indexer and I try to do deduplication where possible, and the reality is that:

    domain.com/this-page-here

can serve entirely different content from

    domain.com/this-page-here/

depending on the server (and application) configuration.

Pretty much the only way to 100% reliably deduplicate URLs is to look at their content, and somehow magically compare content that can change from page load to page load -- which is a whole other problem.