Live data from Hacker News

My history of (mostly failed) side projects and startups

gabrielweinberg.com

71–80 of 82 posts

Re: My history of (mostly failed) side projects and startups

#71

Earlier quoted context omitted.

It varies for different cases depending on a lot of factors like severity, impact on users, etc. In the particular case from above, to find out the history of what might have happened, I just picked a domain at random and dug into its history to find the autogenerated pages with tons of typos for each domain. I kinda thought one example would make the point. Does it help that much more to give another example? I can…

So, basically what you're saying is I went wrong with the typos? I got really excited by my algo and was overzealous with adding it. I believe I did take it off of the sites I issued re-inclusion requests for, but they never got re-included and I never got any messages back (to my knowledge). Also, they were not on every one of those domains. Each site took a long time to make actually. They either involved generatin…

The typos were definitely going overboard. I can understand the appeal of "I've got this great tool--what can I do with it?" But we get a lot of complaints about typo spam, so that's a sensitive issue. I definitely would have done less of that.

There's also a class of folks we call navigation spammers who try to show up for tons of domain name queries. I can give you some history to provide context. In the old days, when you searched for [myspace.com] we'd show a single result as if someone had done the query [info:myspace.com]. The problem is that people would misspell it and do the query [mypsace.com], and then we'd end up either show no result or (usually) a low-quality typo-squatting url. So we made url queries be a string search, so [myspace.com] would return 10 results. That way if someone misspelled the query, they might get the exact-match bad url at #1, but they'd probably get the right answer somewhere else in the top 10. Overall, the change was a big win, because 10% of our queries are misspelled. But if you're showing 10 results for url queries, now there's an opportunity for spammers to SEO for url queries and get dregs of traffic from the #2 to #10 positions. Now we're getting closer to present-day, so I'll just say we've made algorithmic changes to reduce the impact of that.

But you were hitting a bunch of different factors: tons of typos, specifically for misspelled url queries, autogenerated content, lots of different domain names that looked to have a fair amount of overlap (expireddomainscan.com, registereddomainscan.com, refundeddomainscan.com, etc.). If you were doing this again, I'd recommend fewer domain names and putting more UI/value-add work on the individual domains.

Re: My history of (mostly failed) side projects and startups

#72

Earlier quoted context omitted.

The tricky part is that the math works out something along the lines of there being ~200,000,000 domains and there being ~20,000 Google employees. At a simplistic level that works out to 10,000 domains per Google employee. Which means that even if Google stopped doing everything else and everyone at Google spent all their time talking to webmasters, they'd each have to answer 10,000 peoples' questions about rankings,…

I think the best scalable thing that you could do would be to generate a lot more useful automated warnings via all registered channels. And then have a process you outline somewhere on timelines and how to correct. I think the biggest hassle from the user perspective is it all feels like a black hole and black box. I understand the argument behind keeping it a black box, but it doesn't need to be as much of a blackh…

Matt wrote a blog post about this http://www.mattcutts.com/blog/notifying-webmasters-of-penalt... and published a YouTube video http://www.youtube.com/watch?v=hTI1TSmKmps

Google attempts to determine whether you deserve a warning; the goal is to notify honest folks, without notifying real "bad guy" spammers that they've been caught. Naturally, the algorithm gets it wrong sometimes... detecting wrongdoing is easier than detecting intent.

Were you blacklisted from Google Search or Google AdSense or both? Google AdSense's blacklist policy is totally separate from Google Search; Google AdSense's policy is to blacklist people on suspicion of wrong-doing (guilty unless proven innocent).

Re: My history of (mostly failed) side projects and startups

#73

Earlier quoted context omitted.

I think the best scalable thing that you could do would be to generate a lot more useful automated warnings via all registered channels. And then have a process you outline somewhere on timelines and how to correct. I think the biggest hassle from the user perspective is it all feels like a black hole and black box. I understand the argument behind keeping it a black box, but it doesn't need to be as much of a blackh…

Matt wrote a blog post about this http://www.mattcutts.com/blog/notifying-webmasters-of-penalt... and published a YouTube video http://www.youtube.com/watch?v=hTI1TSmKmps Google attempts to determine whether you deserve a warning; the goal is to notify honest folks, without notifying real "bad guy" spammers that they've been caught. Naturally, the algorithm gets it wrong sometimes... detecting wrongdoing is easier th…

Thx--just Google search actually.

Re: My history of (mostly failed) side projects and startups

#74

Why did Google blacklist all of your Tldscan sites? Was it just because your sites' content was updated automatically? Or was it because you did something wrong for SEO?

Maybe this can help . . . I have a few websites that automatically make new posts. As of 10/14, they all show 0 pages indexed in Google. Previously they would get a few thousand visitors per day. I guess Google feels as though they violate their terms and removed them. It seems to me it was a manual removal. I received no emails in webmaster tools about the removal.

"I have a few websites that automatically make new posts."

Making a bunch of autogenerated sites has its risks. For example, if you were just taking a bunch of MP3 names or Hot Trends queries and then scraping twitter for mentions of those phrases and slapping that all up on a website with scripts, that tends to cruft up our index with autogenerated content that users complain about and that violates our quality guidelines. Likewise, if all you were doing was scraping Twitter for phrases like sad or heartbroken or heartless and throwing that scraped Twitter content up on a webpage with a script, users would also complain about that autogenerated content and it would violate our guidelines. Would that be helpful insight?

Re: My history of (mostly failed) side projects and startups

#75

Earlier quoted context omitted.

Maybe this can help . . . I have a few websites that automatically make new posts. As of 10/14, they all show 0 pages indexed in Google. Previously they would get a few thousand visitors per day. I guess Google feels as though they violate their terms and removed them. It seems to me it was a manual removal. I received no emails in webmaster tools about the removal.

"I have a few websites that automatically make new posts." Making a bunch of autogenerated sites has its risks. For example, if you were just taking a bunch of MP3 names or Hot Trends queries and then scraping twitter for mentions of those phrases and slapping that all up on a website with scripts, that tends to cruft up our index with autogenerated content that users complain about and that violates our quality guid…

As a concrete case to discuss, what about something like http://poeet.com/

I made this over a weekend. And the people whose poetry is being captured love it. But it is auto-generated in the sense you're talking about.

It actually went down for a bit and I got a bunch of complaints, enough that I got it back up relatively quickly.

Re: My history of (mostly failed) side projects and startups

#76

Earlier quoted context omitted.

Maybe this can help . . . I have a few websites that automatically make new posts. As of 10/14, they all show 0 pages indexed in Google. Previously they would get a few thousand visitors per day. I guess Google feels as though they violate their terms and removed them. It seems to me it was a manual removal. I received no emails in webmaster tools about the removal.

"I have a few websites that automatically make new posts." Making a bunch of autogenerated sites has its risks. For example, if you were just taking a bunch of MP3 names or Hot Trends queries and then scraping twitter for mentions of those phrases and slapping that all up on a website with scripts, that tends to cruft up our index with autogenerated content that users complain about and that violates our quality guid…

Matt - I agree autogenerated content is a problem and is polluting the search results so I'm glad you guys have taken action. But what about sites like bibleknowledgebookstore.com and articlesubmissionreview.com that are buying links, creating fake content, and spamming web 2.0 profile pages and forums? How come tactics like these are not only working, but dominating competitive markets? What's the point in going after high quality editorial links when sites are rewarded for essentially spamming?

Re: My history of (mostly failed) side projects and startups

#77

Earlier quoted context omitted.

"I have a few websites that automatically make new posts." Making a bunch of autogenerated sites has its risks. For example, if you were just taking a bunch of MP3 names or Hot Trends queries and then scraping twitter for mentions of those phrases and slapping that all up on a website with scripts, that tends to cruft up our index with autogenerated content that users complain about and that violates our quality guid…

As a concrete case to discuss, what about something like http://poeet.com/ I made this over a weekend. And the people whose poetry is being captured love it. But it is auto-generated in the sense you're talking about. It actually went down for a bit and I got a bunch of complaints, enough that I got it back up relatively quickly.

Theres a clear difference between your site http://poeet.com and a clean cut case of auto generated spam. Your site is actually quite creative where it is aggregating content from a twitter hashtag and indexing short poems that may otherwise go unnoticed, you are also showing the users original tweet and @user and not manipulating anything. The original poster was likely scraping content, not providing citation and for the means of having the duplicate content wrapped around ads.

Re: My history of (mostly failed) side projects and startups

#78

Earlier quoted context omitted.

"I have a few websites that automatically make new posts." Making a bunch of autogenerated sites has its risks. For example, if you were just taking a bunch of MP3 names or Hot Trends queries and then scraping twitter for mentions of those phrases and slapping that all up on a website with scripts, that tends to cruft up our index with autogenerated content that users complain about and that violates our quality guid…

Matt - I agree autogenerated content is a problem and is polluting the search results so I'm glad you guys have taken action. But what about sites like bibleknowledgebookstore.com and articlesubmissionreview.com that are buying links, creating fake content, and spamming web 2.0 profile pages and forums? How come tactics like these are not only working, but dominating competitive markets? What's the point in going aft…

Don't forget Google needs content publishers for Adsense. Surely that is the only reason brain liquifying content mills like ehow don't get the slap down? This junk is ridiculous (and this was one of the first pages I looked at):

http://www.ehow.com/blended-families/

How to Plan a Happy Blended Family How to Harmony in Your New Blended Family How to have harmony in your new Blended Family How to Achieve Harmony in a Blended Family How to Nurture A Blended Family How to Successfully Manage a Blended Family

WTF is this junk? Why does ehow.com get 3 million Google visitors a day? The mind boggles!

Re: My history of (mostly failed) side projects and startups

#79
post #78

Earlier quoted context omitted.

Matt - I agree autogenerated content is a problem and is polluting the search results so I'm glad you guys have taken action. But what about sites like bibleknowledgebookstore.com and articlesubmissionreview.com that are buying links, creating fake content, and spamming web 2.0 profile pages and forums? How come tactics like these are not only working, but dominating competitive markets? What's the point in going aft…

Don't forget Google needs content publishers for Adsense. Surely that is the only reason brain liquifying content mills like ehow don't get the slap down? This junk is ridiculous (and this was one of the first pages I looked at): http://www.ehow.com/blended-families/ How to Plan a Happy Blended Family How to Harmony in Your New Blended Family How to have harmony in your new Blended Family How to Achieve Harmony in a…

Great question, though ehow is created by user submission and paid article writers not an individual scraping other users content, publishing it, and not linking back IE $100 plagiarism. I do agree though that eHow is PURE junk and nothing but a site to generate ad revenue. I am not sure if they offer users who submit articles any profit sharing but they are being jipped as well. eHow is by the people behind Enom and a few other networks who give Google a ton of dough for advertising.

Re: My history of (mostly failed) side projects and startups

#80

Earlier quoted context omitted.

Maybe this can help . . . I have a few websites that automatically make new posts. As of 10/14, they all show 0 pages indexed in Google. Previously they would get a few thousand visitors per day. I guess Google feels as though they violate their terms and removed them. It seems to me it was a manual removal. I received no emails in webmaster tools about the removal.

"I have a few websites that automatically make new posts." Making a bunch of autogenerated sites has its risks. For example, if you were just taking a bunch of MP3 names or Hot Trends queries and then scraping twitter for mentions of those phrases and slapping that all up on a website with scripts, that tends to cruft up our index with autogenerated content that users complain about and that violates our quality guid…

Matt, you completely rock. Those are fine examples. :)
Post reply on HN