How My Popular Site was Banned by Google • Hacker News Board

Re: How My Popular Site was Banned by Google

#51

Post by Matt_Cutts » Tue, Oct 11, 2011, 10:23 PM UTC

gojomo wrote: post #44

Matt_Cutts wrote: post #36

Earlier quoted context omitted.

I'm not saying that the issues aren't fixable. But when the site came up for a review, we saw things that violate our quality guidelines: autogenerated pages with hundreds of affiliate links that consist of lots of keywords, and the links/keywords are duplicate content. If he fixes the issues by e.g. blocking out the autogenerated pages, then I expect his homepage will be found in Google. The autogenerated pages are…

Fair enough! Was there a way Brower could have received this clear explanation without appearing on the front page of HN?

The primary ways to get feedback are:

- looking for messages in our webmaster console at google.com/webmasters

- asking in our webmaster forum, also linked to from google.com/webmasters

- doing a reconsideration request (also helps with confirmation of manual action)

- you can talk to search engine reps at various search conferences, e.g. I'll be at PubCon next month.

- calling AdWords support (only for AdWords customers, and this won't give you SEO advice)

We've also been experimenting with 1:1 support over email, by way of a link in our webmaster console. The tension there is finding a solution that scales. We do try to keep an eye on tweets, blog posts, Google+, Hacker News, and similar places around the web, but that's also hard to scale.

Re: How My Popular Site was Banned by Google

#52

Post by joshu » Tue, Oct 11, 2011, 10:26 PM UTC

droithomme wrote: post #29

Matt_Cutts wrote: post #15

You have an autogenerated web site that consists of practically nothing other than affiliate links to Amazon. You can make an infinite number of autogenerated pages on your site, e.g. http://www.filleritem.com/index.html?q=hacker+news http://www.filleritem.com/index.html?q=31.69 http://www.filleritem.com/index.html?q=teen+sex and each autogenerated page consists of literally hundreds of affiliate links stuffed with k…

I have no association with this person and just heard about this. But obviously those are search term result pages and not "autogenerated pages" in the usual sense. If Google is going to apply such a standard uniformly, they'd have to ban all sites that have search functions. Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 diffe…

Search results are also not supposed to be indexed. I believe this is actually in the guidelines.

Re: How My Popular Site was Banned by Google

#53

Post by hbar » Tue, Oct 11, 2011, 10:36 PM UTC

Matt_Cutts wrote: post #15

You have an autogenerated web site that consists of practically nothing other than affiliate links to Amazon. You can make an infinite number of autogenerated pages on your site, e.g. http://www.filleritem.com/index.html?q=hacker+news http://www.filleritem.com/index.html?q=31.69 http://www.filleritem.com/index.html?q=teen+sex and each autogenerated page consists of literally hundreds of affiliate links stuffed with k…

The same could be said of Google's search engine itself: an autogenerated web site consisting of essentially nothing other than links to other places, many of them paid. But no one can deny that it's a useful site.

I think that is the issue here -- probably the vast majority of such sites are undesirable, but you can't discount them all.

Re: How My Popular Site was Banned by Google

#54

Post by Matt_Cutts » Tue, Oct 11, 2011, 10:38 PM UTC

joshu wrote: post #52

droithomme wrote: post #29

Earlier quoted context omitted.

I have no association with this person and just heard about this. But obviously those are search term result pages and not "autogenerated pages" in the usual sense. If Google is going to apply such a standard uniformly, they'd have to ban all sites that have search functions. Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 diffe…

Search results are also not supposed to be indexed. I believe this is actually in the guidelines.

Yup. Search for [quality guidelines] and they're at http://www.google.com/support/webmasters/bin/answer.py?answe... . The relevant part is "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."

Re: How My Popular Site was Banned by Google

#55

Post by Matt_Cutts » Tue, Oct 11, 2011, 10:45 PM UTC

OpenAlgorithm wrote: post #41

Matt_Cutts wrote: post #39

Earlier quoted context omitted.

When I do the search [filler item], one of the top results I see is http://www.makeuseof.com/tag/top-5-amazon-filler-item-finder... which shows five different services to fill this information need, and that also has pictures. I do think that's a pretty helpful result. I mentioned filleritemfinder.com as a random example (there are many of these services), but filleritemfinder.com appears to use AJAX to keep results…

Would it be likely that if the site was to use another monetization plan that it would be less likely to be penalized e.g. advertising instead of affiliate links?

I recently talked for about a minute about the topic of "too much advertising" that sometimes drowns out the content of a page. It was in this question and answer session that we streamed live on YouTube: http://www.youtube.com/watch?v=R7Yv6DzHBvE#t=19m25s . The short answer is that we have been looking more at this. We care when a page creates a low-quality or frustrating user experience, regardless of whether it's because of affiliate links, advertising, or something else.

Re: How My Popular Site was Banned by Google

#56

Post by thenextcorner » Tue, Oct 11, 2011, 10:45 PM UTC

joshu wrote: post #52

droithomme wrote: post #29

Earlier quoted context omitted.

I have no association with this person and just heard about this. But obviously those are search term result pages and not "autogenerated pages" in the usual sense. If Google is going to apply such a standard uniformly, they'd have to ban all sites that have search functions. Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 diffe…

Search results are also not supposed to be indexed. I believe this is actually in the guidelines.

Literally Google Webmaster Guidelines say: "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."

So a couple of questions: - What is of value for a user? - Who is determining the value for the users in these cases?

As always, it's not always quite clear on what treatment you should use for search pages!

Re: How My Popular Site was Banned by Google

#57

Post by robryan » Tue, Oct 11, 2011, 10:50 PM UTC

Matt_Cutts wrote: post #51

gojomo wrote: post #44

Earlier quoted context omitted.

Fair enough! Was there a way Brower could have received this clear explanation without appearing on the front page of HN?

The primary ways to get feedback are: - looking for messages in our webmaster console at google.com/webmasters - asking in our webmaster forum, also linked to from google.com/webmasters - doing a reconsideration request (also helps with confirmation of manual action) - you can talk to search engine reps at various search conferences, e.g. I'll be at PubCon next month. - calling AdWords support (only for AdWords custo…

I don't think it is really possible here to have a solution that provides a good outcome for all users and also scales. For every person who does a proper write up and gets some attention places like here there would no doubt be many others who have had the same issues but got no attention.

What would really help is a clear reasoning given for the ban and a set of steps a site can take to get back in. From what I see often web masters are more than willing to make changes, they just don't know which to make. Often to the bans are said to be final, denying someone the chance to make it better.

I guess it is tempting to provide minimal support for organic search as people aren't paying and generally there are plenty of other sites to cover ones absence. One thing that would be good is say a yearly fee which guarantees that you can get one on one support if something happens.

Re: How My Popular Site was Banned by Google

#58

Post by robryan » Tue, Oct 11, 2011, 10:59 PM UTC

joshu wrote: post #52

droithomme wrote: post #29

Earlier quoted context omitted.

I have no association with this person and just heard about this. But obviously those are search term result pages and not "autogenerated pages" in the usual sense. If Google is going to apply such a standard uniformly, they'd have to ban all sites that have search functions. Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 diffe…

Search results are also not supposed to be indexed. I believe this is actually in the guidelines.

It's hard to draw the line on what is search and what isn't. If you use a system where the last part of the URL will be treated as a search but only certain ones are ever linked to so they are a product page.

Also if you have some kind of recent searches list and those link.

Re: How My Popular Site was Banned by Google

#59

Post by ceejayoz » Tue, Oct 11, 2011, 11:03 PM UTC

StrawberryFrog wrote: post #37

chime wrote: post #35

Earlier quoted context omitted.

> Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 different possibilities Actually there is no difference between search page result sites like the OP and what you said. Having a URL that ends in .html does not mean there is a static html file on the server. For examples, nextag and pricegrabber have URLs like blah.com/digital/C…

> Actually there is no difference between search page result sites like the OP and what you said Uh, yes there is. http://www.somesite.com/index.html?q=foo is obviously a search (right down to the choice if q for query), but http://www.somesite.com/buy_foo_now.html does not look like one. You might say that technically they could have the same back-end, but the web is flexible - just about any URL scheme could resolv…

> Uh, yes there is. http://www.somesite.com/index.html?q=foo is obviously a search

Obviously? Drupal's default non-rewritten URLs are index.php?q=foo, where foo is the page path. `q` can and does stand for more than one thing.

Re: How My Popular Site was Banned by Google

#60

Post by robryan » Tue, Oct 11, 2011, 11:05 PM UTC

chime wrote: post #35

droithomme wrote: post #29

Earlier quoted context omitted.

I have no association with this person and just heard about this. But obviously those are search term result pages and not "autogenerated pages" in the usual sense. If Google is going to apply such a standard uniformly, they'd have to ban all sites that have search functions. Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 diffe…

> Autogenerated in the bad sense is link farms like fakesite.com/buy_drugwiththisname_now.html, with drugwiththisname replaced with 500,000 different possibilities Actually there is no difference between search page result sites like the OP and what you said. Having a URL that ends in .html does not mean there is a static html file on the server. For examples, nextag and pricegrabber have URLs like blah.com/digital/C…

Would be easy for the google bot to work this out to, replace the URL part that looks like a dynamic variable with different values and see if it returns a page for them all like a search engine would. Surprised it doesn't appear to do this.