Ask HN: What do you do when your website seems to be penalised by Google?

Exactly. It's so difficult to get URLs "right", and that's quite non-obvious until you do something like writing a crawler.

Another example is whether foo.com/bar is the same as foo.com/BAR. Usually yes, but it's entirely possible that they will serve different content.

Also, which URL parameters should be disregarded, and which should be considered important? A crawler must do quite a bit of nontrivial page introspection in order to figure out the answer to that all on its own.

Often pages that are essentially the same will be a bit different. Timestamps and time-sensitive data (eg. listings on a marketplace) will trip you up, here.

hi, first things first

do not use the word "penalty" - you want to show up for a certain query in google, you think you should show up for it, you don't show up for it.

that is the issue, nothing else.

you formulated a hypothesis: you think your site is "penalised by Google"

ok, go to google webmaster tools and verify

  - http://blog.mavnn.co.uk/
  - http://mavnn.co.uk/
  - http://www.mavnn.co.uk/

check the "Site Messages" navigation point of all these domain variations, if you have a penalty, then there will be a message. pro tip: only ever talk about "penality" if you get a message that says you have a "penality". (everything else is just SEO b#llshit talk)

my guess: there won't be such a message.

ok, the second quess is the wrong canonical, you already fixed that one. but: if you point a canonical to an HTTP 301 redirect, and the redirect points back to the original URL google will basically ignore the canonical. the canonical could have been the issue, but as https://www.google.com/search?q=site%3Ablog.mavnn.co.uk%2Fty... has been indexed (without ending slash) i doubt it.

ok, let's look at anything that might be unusual about your site

i.e.: your start page http://www.mavnn.co.uk/

basically it consists out of a "Hello World" and a link to a broken URL and a link to a piece of duplicated text.

"Hello World" is a typical "this server was just set up, nothing to see here" message.

your start page is not indexed (see: site:http://www.mavnn.co.uk/ )

that is strange. let's formulate a hypothesis.

your startpage communicated a basic "this server has just been set up, nothing to see here" message. google has a) no interrest into indexing such websites b) the webmasters are pretty pissed, if their newly set up servers are indexed in this way, as newly set up servers are usually not very secure, yet

additional google sees common subdomains i.e. blog.example.com as part of the main site and not as independent webproperties (yeah, they figured that one out quite some time ago).

hypothesis: you communicate via your startsite that your website is not yet - probably - set up and that is why it does not send you traffic.

my bet is, that this is the case. why? because you startpage is the one thing that is definitely not ... like other websites our there.

fix it, two possibilities: http://www.mavnn.co.uk/ -> HTTP 301 -> http://blog.mavnn.co.uk/

or you set up a proper startpage, some text what this is, some links to your other ressources.

after you have done one of this, do a fetch-as-googlebot (via google webmaster tools) and click the "submit to index" button.

wait two days.

if not, test another hypothesis or post in the google webmaster forum, actually google guys dig these kind of errors.

Regarding the spam sites, in your RSS feed, you are publishing your full articles. More than likely, the scraper sites are pulling directly from these feeds, publishing quickly and getting Googlebot to see the content before it hits your site (thus receiving attribution). I would suggest:

1) Summaries only in RSS feeds. 2) Throttle the RSS feed back by several hours so that your latest article is not listed immediately. 3) Upon publishing, immediately link to the article via all of your social media outlets. 4) When internally linking within articles, use full URL paths and not relative. (If the spam sites are directly pulling your content and not cleaning up, you may be able to get a link back to your site from the scraped content.)

When publishing, timing is everything. Just my $0.02 based on my own experiences dealing with spam sites.

On a side note, even though we are in the age of HTML5, I would still suggest sticking with one H1 tag per page, if possible.

This sucks. I'm not saying it's not the answer, but the fact that you have to castrate your feed because spam sites can actually get "SEO credit" for your content just sucks. I always loved RSS feeds that published the full text, because I could read whole articles without having to click through.

Semantic web could fix this a little by making it easier to scrape with the tag, but publishing content is exactly what RSS was meant to do.

I wish Google would (if even possible) find a better way to fix this. In the same way that there's an actual argument against single page apps because "they can't be indexed" or "SEO, man." Discoverability shouldn't be holding back progress (in an ideal world, I know). Rather, indexing should adapt to new technology so that we can make a better web that's still discoverable by users.

I agree, but with sites that do not have much authority (aka PageRank), it's difficult to determine attribution when scraped content is coming online just as quickly as an original post. Googlebot will generally hit a site several times a day, but if it's hitting the spammer's site first or if the spammer site has more authority, it's a long uphill climb to get things turned around.

I should also point out that this is just one thing to consider amongst the other points already made by others.