Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How My Popular Site was Banned by Google

kbrower.posterous.com

Post reply on HN
Print view View original

81–90 of 132 posts

Re: How My Popular Site was Banned by Google

#81

Post by Matt_Cutts »

dquigley wrote: post #63
Matt_Cutts wrote: post #51

Earlier quoted context omitted.

The primary ways to get feedback are: - looking for messages in our webmaster console at google.com/webmasters - asking in our webmaster forum, also linked to from google.com/webmasters - doing a reconsideration request (also helps with confirmation of manual action) - you can talk to search engine reps at various search conferences, e.g. I'll be at PubCon next month. - calling AdWords support (only for AdWords custo…

Matt, Would it be possible/financially practically for Google to offer a premium support option that would pay for itself? Figure out how much it costs to respond to someone, offer that as a flat fee with a clear indication, that paying only gets you a consult on what's wrong with your site, and not any promise that it will get you reinstated. Basically what you did in your comments above. I would personally make it…

It would be possible. I think Google has a bit of a historical aversion to it because it reminds us of pay-for-inclusion, in which someone would pay for their web page to be crawled/indexed.

Larry Page was especially critical of pay-for-inclusion because it skews your incentives: if you don't crawl the web well, then people pay you to fix your own shortcomings, which in turn encourages you to have more shortcomings.

I think Google also comes from the perspective of self-service AdWords being successful, so the idea of self-service (free) diagnostics really appeals to us. That's why we've put a lot of effort into our free webmaster tools.

I wouldn't be philosophically opposed to a pay-for-support system if it were done well, but it would be a tricky thing to get right. Normally when we consider it, we end up saying things like "Why don't we just try to make it so that people don't need that option?"

Re: How My Popular Site was Banned by Google

#82

Post by Matt_Cutts »

kposehn wrote: post #66
Matt_Cutts wrote: post #51

Earlier quoted context omitted.

The primary ways to get feedback are: - looking for messages in our webmaster console at google.com/webmasters - asking in our webmaster forum, also linked to from google.com/webmasters - doing a reconsideration request (also helps with confirmation of manual action) - you can talk to search engine reps at various search conferences, e.g. I'll be at PubCon next month. - calling AdWords support (only for AdWords custo…

Hi Matt, thanks for chiming on on this thread. From your reply, I understand that your primary issue is indeed with scaling any manner of support issues. As I understand it, the current thinking at Google is to seek automated solutions over technical to reduce costs and improve issue volume handling. With that in mind, is there a system of triggers that can flag issues for manual review or intervention instead of bei…

"With that in mind, is there a system of triggers that can flag issues for manual review or intervention instead of being handled completely automatically?"

I can't speak for the ads folks at all, unfortunately, but on the websearch side we certainly strive to do as much as we can algorithmically, but also to use our manual cycles effectively. The site in question was flagged algorithmically, but also sent to a member of the manual webspam team who concluded that it violated our quality guidelines. When a reconsideration request came in, it went back to a manual person for review again.

To the extent we can figure out ways to do it without compromising our systems, I think both websearch and ads would like to be as transparent as we can.

Re: How My Popular Site was Banned by Google

#83

Post by Matt_Cutts »

option1138 wrote: post #65
Matt_Cutts wrote: post #51

Earlier quoted context omitted.

The primary ways to get feedback are: - looking for messages in our webmaster console at google.com/webmasters - asking in our webmaster forum, also linked to from google.com/webmasters - doing a reconsideration request (also helps with confirmation of manual action) - you can talk to search engine reps at various search conferences, e.g. I'll be at PubCon next month. - calling AdWords support (only for AdWords custo…

Hi Matt, We don't know each other but I think we know of each other. I'm rather immersed in webspam detection and found this incredibly interesting. You imply that the challenge is finding a solution that scales. Yet it sounds to me from your response that this site was flagged via manual review. Did I misunderstand? If I heard you correctly, then is manual review a significant equation in the webspam detection metho…

The site was flagged both algorithmically and also escalated to a member of the manual webspam team.

The basic philosophy is to do as much as we can algorithmically, but there will always be a residual of hard cases that computers might not do as well at (e.g. spotting hacked sites and identifying the parts of a site that have been hacked). That's where the manual webspam team really adds a lot of value.

In addition to things like removing sites, the data from the manual webspam team is also used to train the next generations of our algorithms. For example, the hacked site data that our manual team produced not only helped webmasters directly, we also used that data to produce an automatic hacked site detector.

If you're interested, I made a video about the interaction between algorithmic and manual spamfighting here: http://www.youtube.com/watch?v=ES01L4xjSXE

Re: How My Popular Site was Banned by Google

#84

Post by Iv »

mapgrep wrote: post #74
modeless wrote: post #73

Earlier quoted context omitted.

The difference is Google uses robots.txt to exclude these autogenerated pages from search engine crawlers. filleritem.com didn't; now they do[1]. Quite simple really. [1]: http://news.ycombinator.com/item?id=3100488

No one should be banned from Google for failing to fill out robots.txt in the manner Google prefers. This is a horrible abuse of power -- and a competitor -- by Google. "Quite simple really."

Even if I generally like Google, we do need a competitor. I occasionally use duckduckgo

Re: How My Popular Site was Banned by Google

#85

Post by Matt_Cutts »

eurohacker wrote: post #77
Matt_Cutts wrote: post #15

You have an autogenerated web site that consists of practically nothing other than affiliate links to Amazon. You can make an infinite number of autogenerated pages on your site, e.g. http://www.filleritem.com/index.html?q=hacker+news http://www.filleritem.com/index.html?q=31.69 http://www.filleritem.com/index.html?q=teen+sex and each autogenerated page consists of literally hundreds of affiliate links stuffed with k…

As we have heard lately how Google has manipulated the Yelp search results and others for their benefit, one has to wonder - since google has aquired some big websites like Zagat, theoretically - if one day they will just start to screw Zagat competitors , what can be actually done about it , i mean how can one prove that google is just attacking you and they just dont like you as their competitor ... its relative -…

It's a fair question, but our quality guidelines have been quite consistent for the last decade or so. Our team takes action without regard to whether a website is an advertiser, partner, or competitor. The long-term loyalty/trust of our users is worth much more than any sort of short-term revenue that a sneaky deal would provide. And Google's culture is such that anyone inside or outside of the company can claim that a particular practice isn't in line with "Don't be evil" and kick off a fair amount of self-scrutiny.

Ultimately though if you don't trust Google, we don't lock your data in, so you can use a different search engine or service just by entering it in the address bar. That's the ultimate check on Google: if we start to act too abusive or "evil" we know that people can desert us. So it's in our enlightened self-interest to try to act in our users' long-term interests.

Re: How My Popular Site was Banned by Google

#86

Post by relix »

Matt_Cutts wrote: post #80
algoshift wrote: post #76

Earlier quoted context omitted.

Matt, I think the main problem (having seen what it looks and feels like on the receiving end) is that Google's solution and treatment of the site owner is, for lack of a better term, rather violent. Put yourself in the shoes of someone who is not a scammer and is trying to do a good job. Perhaps someone how is learning the ropes. Getting hit with an unilateral, violent and --for all intents and purposes-- permanent…

I take your points, and in my group we've worked hard in the last year to improve our transparency and our appeals process, e.g. see http://googlewebmastercentral.blogspot.com/2011/09/reconside... from a few weeks ago. Likewise, AdWords now has phone support for all AdWords advertisers. I can't speak for the AdSense (for Domains) team other than to say that when they shut down an account, they think that they have go…

Support for all Google users would obviously be painful.

I think it would be fair if there was some personal support for people who either spend a lot of money on Adwords (I think there already is?) or make a lot of money on Adsense. These people probably have the highest levels of stress due to "insta-evaporation probability", and no doubt the amount of support people would be several magnitudes less if you used a threshold like that.

Re: How My Popular Site was Banned by Google

#87

Post by davidwhitehouse »

general-marty wrote: post #62

University should offer courses on Google policy. A thorough understanding of such policies will be the primary determining factor for your success on the web.

I disagree with you there general-marty, I think the primary factor determining your success on the web is a decent business plan and marketing strategy. Without that you rely on free traffic from Google. Any company that puts all its eggs in one basket (e.g. Google Organic) is skating on thin ice.

Re: How My Popular Site was Banned by Google

#88

Post by ErrantX »

mapgrep wrote: post #74
modeless wrote: post #73

Earlier quoted context omitted.

The difference is Google uses robots.txt to exclude these autogenerated pages from search engine crawlers. filleritem.com didn't; now they do[1]. Quite simple really. [1]: http://news.ycombinator.com/item?id=3100488

No one should be banned from Google for failing to fill out robots.txt in the manner Google prefers. This is a horrible abuse of power -- and a competitor -- by Google. "Quite simple really."

I disagree; Google should have strong rules to filter out crap and give me great search results.

In this case we have something of an edge case; the site is useful, but the auto-generated pages cause potential problems for search results.

If you were talking about how Google communicates these issues, or how it goes about resolving these edge cases, then I'd be agreeing. Much to be improved.

But getting rid of the potential issue.. thumbs up.

Re: How My Popular Site was Banned by Google

#89

Post by SteveOllington »

lol, poor Matt, always under fire from everybody who hasn't got the rankings they want. It's simple really, Google try to make it fair for everybody (and granted it doesn't always work perfectly, which is why the continue to make changes, it's evolution) and if some sites give themselves an unfair advantage and are caught, then they're punished. How else could a search engine work? If it were any different then al of our results would be greed pages with no useful info and everybody would stop Googling stuff.

As somebody once quoted, "You can please some of the people, some of the time, but you can't please all of the people, all of the time!"

Re: How My Popular Site was Banned by Google

#90

Post by silverbax88 »

mapgrep wrote: post #70
Matt_Cutts wrote: post #36

Earlier quoted context omitted.

I'm not saying that the issues aren't fixable. But when the site came up for a review, we saw things that violate our quality guidelines: autogenerated pages with hundreds of affiliate links that consist of lots of keywords, and the links/keywords are duplicate content. If he fixes the issues by e.g. blocking out the autogenerated pages, then I expect his homepage will be found in Google. The autogenerated pages are…

This is moronic. The pages are "autogenerated" because it's a web app. They have affiliate links because that MAKES TOTAL SENSE given the nature of the app ("show me as many products as possible that cost $X"). You know who else generates infinite autogenerated pages with affiliate codes attached? GOOGLE. http://www.google.com/search?q=teen+sex&tbm=shop&hl=... It's scary that Google uses "logic" like yours to make th…

Agreed. There are no issues with this site as it works today. Google is way off base with this and it's troublesome.
Post reply on HN