Live data from Hacker News

How My Popular Site was Banned by Google

kbrower.posterous.com

121–130 of 132 posts

Re: How My Popular Site was Banned by Google

#121

Amazon probably complained to Google and Google despite already having the rules on the books decided to enforce it on this one site, while also keeping many other low traffic clone sites offering the same service alone. Call me paranoid, or cynical but I've seen much worse from Google on the AdWords side of their offerings.

Nope, that didn't happen.

This has been an amazing thread but there really is a pattern that is hard to follow for webmasters. In the case of the original site couldn't Google have just de-indexed the "extra" pages but left the the index page alone since it is indeed useful?

Also, how did his site get canned but others like sportslinkup.com, which is an ebay affiliate spam site cloaked as a link directory, have over 7 million indexed pages and 8 million indexed images (all hosted by ebay, not the sports site) and it gets over half a million Google visitors per month (according to compete.com)?!? I think the sports site is even scraping Google for keywords, it's full of examples of what not to do, but it's been sailing along for years with Google approval, or at least no automated detection.

The fine line between sites getting canned and sites getting MASSIVE traffic for essentially the same thing is very confusing.

Re: How My Popular Site was Banned by Google

#122
post #79

Earlier quoted context omitted.

Yup. Search for [quality guidelines] and they're at http://www.google.com/support/webmasters/bin/answer.py?answe... . The relevant part is "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines."

Matt, I tried that recently with one of my own sites, and after a few days I got a warning in my Google Webmaster Tools dashboard that the bot could not access my /search URL because it was blocked in my robots.txt, and I should take action to correct it. So I then unlocked my /search URL, which is in violation of the above guideline, but made the error in Google Webmaster Tools go away. These conflicting messages fr…

Hi j_col-

In that case, Google Webmaster tools is not actually reporting an error. That's a report to show you what URLs Google tried to crawl but couldn't (due to being blocked) so you can review it and ensure that you are not accidentally blocking URLs that you want to have indexed.

I agree that it's confusing in that the report is in the "crawl errors" section.

(I built Google webmaster tools so this confusion is entirely my fault; but I don't work at Google anymore so sadly I can't fix this.)

Re: How My Popular Site was Banned by Google

#123

Earlier quoted context omitted.

As we have heard lately how Google has manipulated the Yelp search results and others for their benefit, one has to wonder - since google has aquired some big websites like Zagat, theoretically - if one day they will just start to screw Zagat competitors , what can be actually done about it , i mean how can one prove that google is just attacking you and they just dont like you as their competitor ... its relative -…

It's a fair question, but our quality guidelines have been quite consistent for the last decade or so. Our team takes action without regard to whether a website is an advertiser, partner, or competitor. The long-term loyalty/trust of our users is worth much more than any sort of short-term revenue that a sneaky deal would provide. And Google's culture is such that anyone inside or outside of the company can claim tha…

Google is a "Habit" now!

Like Google, webmasters like us also try to improve quality of their web pages, but sometimes we fail to understand what actually "Quality" means to Google. May be the angle from your see, is not easy for us to catch. It would be better if Google can publish some Dos and Don't list for the Panda.

For a keyword like "dating", only top companies are being given space on the first page. Well, if you see dating.com, they have nothing like dating in their site and also the articles are nothing but crap, but they managed to be on top and survived in Panda, while small fish like us are no where. Having a far better site, useful contents, more than 200000 active users has no use.

You do not expect a "Dating" website to hire 10 content writers and keep posting fresh articles, because those are the "Contents" Google consider as content. All niche can not be put under one "Quality" guideline.

My apology if i said anything wrong.

Re: How My Popular Site was Banned by Google

#124

http://www.filleritem.com/index.html?q=teen+sex This should have no results. Other queries should have relevant results, but they don't. How can anyone complain over his site being axed? The guy needs to code a better site and I sure as hell would never use that so he can get affiliate money. Sorry, but your fortunate this site has survived for so long.

If you look around, almost everything you click on the web is set up to make someone money. It's none of Google's business how a website is making their money as long as it's legal and follows FTC policies. Browsers make money every time you use that little search box in the top right corner of Firefox. People have a right to be compensated for their work. If the site provides a good service, I'll use it. If you don't want people to make money off your clicks. It would be best to close your browser and never open it again.

My site got hit by Panda (no manual penalty). After relaunching my site (which I had luckily been working on for over a year), my bounce rate is 30% and my direct link rate is 50%. Still, Google has me by the throat. My traffic is almost dead. Panda doesn't care if you're solving a problem for users and giving them the eye candy they want. It apparently just wants your site to be a "unique, well-written" news site. People come to my site to browse vintage items and find stuff that they didn't know existed in the niche. They usually visit during work hours and late in the evening. Reading ease for these items is on a college level (Which Panda doesn't like). They don't want to read a 1000 word article, they usually just want to hear it, see it, maybe ask question or two about the item and see who may be selling one. I offer that.

Panda's obvious Bayesian nature doesn't understand nuance. It only knows napalm.

Niche markets are the ones getting the crap end of the stick. And it's funny, niche markets used to be what made the web so great! Not anymore. Now it's 100% Googlized Walmart-ization.

And yeah, I'll be using www.filleritem.com when shopping Amazon. I didn't know it existed until I read this post. I don't mind, transparently, giving someone a few pennies for a good service. You do it all the time.

Re: How My Popular Site was Banned by Google

#125

Earlier quoted context omitted.

Matt, I think the main problem (having seen what it looks and feels like on the receiving end) is that Google's solution and treatment of the site owner is, for lack of a better term, rather violent. Put yourself in the shoes of someone who is not a scammer and is trying to do a good job. Perhaps someone how is learning the ropes. Getting hit with an unilateral, violent and --for all intents and purposes-- permanent…

I take your points, and in my group we've worked hard in the last year to improve our transparency and our appeals process, e.g. see http://googlewebmastercentral.blogspot.com/2011/09/reconside... from a few weeks ago. Likewise, AdWords now has phone support for all AdWords advertisers. I can't speak for the AdSense (for Domains) team other than to say that when they shut down an account, they think that they have go…

Sorry it took a few days to reply to this. Too busy writing code...

With regards to the argument that describes the resources required for customer support even to a small percentage of your audience I can only say this: It's your chosen business model. I am paraphrasing one of my favorite answers when someone complains about their job ("It's your chosen profession").

The point is that Google's business is about doing what it does for a huge number of people. If supporting them is overwhelming either get out of that business or figure out how to do it correctly. I can't really accept the "it's too many people" argument as a valid reason for not doing it well or for applying the "criminal algorithm" to everyone.

There's a side thread there that says that "providing information about how the abuse was flagged would help spammers quite a bit". While true, I, again, find myself not agreeing with the idea of punishing legitimate customers for this reason. I would like to think that the vast majority of Google customers fall under the "legitimate" category. If spammers get better because you are providing detail flagging information you will simply have to get better at detecting and blocking spammers. This would trigger an evolutionary phase which, at some point, should make it very difficult for a spammer to game the system, even with "full source" if you will. Much like security algorithms become more secure if the source is released and is tested with full knowledge of the internals, yours should do the same.

Conversely, honest and legitimate customers would gain the huge benefit of now understanding how to behave or how to do things and why a certain approach might not do well in Google's ecosystem.

One of the most frustrating things I have seen is someone full of drive to launch an internet business only to be shot down by a Google shutdown. And, when no reason or actionable information for the punishment is given this entrepreneur simply had to throw their hands up and give up on that tack. Needless to say, their next attempt ignored Google products completely and they are doing OK. My guess is that they could have done very well and much grief could have been prevented had Google said: "Your site has the following problems ..." and then, rather than cut them off schedule them for a review in, say, thirty days (or variable length based on the seriousness of the issue). That would have been far more civilized and far more conducive to helping your community grow and evolve in the right direction.

New and inexperienced internet entrepreneurs (and some experienced ones) need a way to learn how to behave. What works and what does not. What is acceptable and what isn't. It is only reasonable to assume that they will make many mistakes in their zeal to get an idea off the ground. Penalizing them with a permanent hammer blow to the head is not conducive to growing better netizens. Guiding them with actionable feedback is.

The current process can only be characterized as violent. From the perspective of an honest business person it is tantamount to getting hit with a bullet while walking your dog. The reasons could have been many. Maybe the shooter objected to your walking your dog in front of their home. Had the shooter at least attempted to communicate with the dog-walker it is far more likely that violence could have been averted.

Above all, if your "Do no evil" is sincere, then you have to change the way this works right away. The way this hits honest entrepreneurs is nothing less than pure evil. Again, you take a bullet and you don't know why.

I do appreciate your visibility here in HN. In the past I have simply given up trying to raise these and other points with anyone at Google that might remotely have the ability to at least elevate the conversation internally. I hope you might be that person. I mean all of the above in the vein of constructive criticism. We all want to see the ecosystem becoming more conducive to the exploration of new ideas. Google, at this time, has taken a rather totalitarian position of being the "moral authority", if you will. With that, and so long as you want to be a benevolent dictator, I think, you inherit the responsibility to not cause harm through your actions.

Having said that, until things change I have no choice but to treat your offerings as something that one simply cannot rely on to build a business. The "Google love" can disappear from your site overnight and you'll have no practical way to fix it. That's not a business, that's going to Vegas.

Re: How My Popular Site was Banned by Google

#126
post #79

Earlier quoted context omitted.

Matt, I tried that recently with one of my own sites, and after a few days I got a warning in my Google Webmaster Tools dashboard that the bot could not access my /search URL because it was blocked in my robots.txt, and I should take action to correct it. So I then unlocked my /search URL, which is in violation of the above guideline, but made the error in Google Webmaster Tools go away. These conflicting messages fr…

Hi j_col- In that case, Google Webmaster tools is not actually reporting an error. That's a report to show you what URLs Google tried to crawl but couldn't (due to being blocked) so you can review it and ensure that you are not accidentally blocking URLs that you want to have indexed. I agree that it's confusing in that the report is in the "crawl errors" section. (I built Google webmaster tools so this confusion is…

Thanks for the response, I will block my /search URL once more via robots.txt and will ignore the warnings in the Webmaster Tools.

Re: How My Popular Site was Banned by Google

#127

Earlier quoted context omitted.

Thanks Matt, took a look at the video, that answers my original question. Also is there a preferred monetization model, e.g. do Google think advertisements are more or less harmful to the user experience than affiliate links, sponsored posts, etc? Obviously across different models you can't just track space taken up, so is there some kind of metric that tracks the rate of content diluting via monetization?

I know you were looking for an answer from Matt but I thought I would offer up my opinion here (as you might have noticed, I love talking about this stuff). Our models currently suggest that the presence of contextual advertising is a significant predictive factor of webspam. We use 10-fold bagging and classification trees, so it's not all that easy to generalize. But I pulled one model out at random for fun. The top…

Thanks for your response, is the data you are viewing publicly available?

Re: How My Popular Site was Banned by Google

#128

Earlier quoted context omitted.

I agree but don't you think that from an algorithmic point of view Google would be better looking at what the user wants and what monetization models they prefer to see versus the averages in terms of monetization models on spam sites. That way their focussing less on removing spammers and more on user quality and thus removing spam.

This really made me think. But I took exception to your comment: I agree but don't you think that from an algorithmic point of view Google would be better looking at what the user wants and what monetization models they prefer[...] No. In fact, I am a rather loudmouthed opponent to Google's somewhat clumsy attempts to measure this ala "Quality Score". In addition to webspam detection and machine learning, I have spen…

Well we know Google can classify what spam sites are.

And assuming they could (despite your arguments against) determine the method of monetization on a site, then simply compare the models they see on spam sites versus another metric which would track the user's reaction to certain types of monetization models.

Re: How My Popular Site was Banned by Google

#129

Earlier quoted context omitted.

I know you were looking for an answer from Matt but I thought I would offer up my opinion here (as you might have noticed, I love talking about this stuff). Our models currently suggest that the presence of contextual advertising is a significant predictive factor of webspam. We use 10-fold bagging and classification trees, so it's not all that easy to generalize. But I pulled one model out at random for fun. The top…

Thanks for your response, is the data you are viewing publicly available?

No...

Re: How My Popular Site was Banned by Google

#130

Earlier quoted context omitted.

This really made me think. But I took exception to your comment: I agree but don't you think that from an algorithmic point of view Google would be better looking at what the user wants and what monetization models they prefer[...] No. In fact, I am a rather loudmouthed opponent to Google's somewhat clumsy attempts to measure this ala "Quality Score". In addition to webspam detection and machine learning, I have spen…

Well we know Google can classify what spam sites are. And assuming they could (despite your arguments against) determine the method of monetization on a site, then simply compare the models they see on spam sites versus another metric which would track the user's reaction to certain types of monetization models.

And assuming they could (despite your arguments against) determine the method of monetization on a site

I can assure you that that they cannot do this accurately. You would be amazed at the scummy business models that openly advertise on Google and are not caught. It would be incredibly difficult to do so as some of them are downright ingenious (one example: free software that updates your drivers.)

It sounds to me that you are positing some type of magical technology that doesn't exist predicated on Google's seeming omniscience. Of course, I am eager to stand corrected...?

Post reply on HN