Search engines and SEO spam

That may be true, but I think one of the good points made on the OP is that it might actually be cultural constraints that keep them from solving the problem:

https://twitter.com/paulg/status/1477761335412809729:

> You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant).

Google has some very smart and knowledgeable people, but the things they do have to fit into certain boxes, which means there are some problems they just can't fix, e.g.

* Everything has to be automated at scale, which leads to consistent poor user experience (unappealable account closures initiated by inscrutable algorithms, SEO spam).

* You get promoted by building new products, not maintaining existing ones, which leads to self-defeating churn outside of core areas (e.g. abandoning Google Talk and squandering their position in the messenger market).

* etc.

All the various suggestions in this thread plus far more complex and insightful solutions are known to Google. Most of it boils down to using automated user feedback to improve or measure search result relevancy.

Google doesn't need to solicit user upvotes / downvotes to improve rankings. They can monitor user clicks on results in addition to analytics on the sites the users visit to determine which sites are relevant to which searches.

Google doesn't optimize for search relevancy.

When I run web sites I frequently look at the log and find a large fraction of the traffic is from search engines. This is a problem because it costs me money to serve that traffic. It might not be initially obvious but it costs more than serving real users because the search engines will scan everything and break the cache.

Google sends a significant amount of traffic. Bing sends a detectable amount of traffic. Baidu's crawler might be more active than the two of those together but I never get hits from Baidu. Other crawlers deliver me trouble instead of value: even if I'm not interested in hosting pirate or plagiarized content, a crawler that is looking for trouble is only going to bring me trouble.

I hate doing it but I turn off crawlers other than Google and Bing both at the robots.txt and web server level because I just can't afford to serve Baidu queries.

I'd like to sign an exclusivity contract with a search engine such that they get exclusive access to crawl it and in turn I get a privileged position in search results. This would give the search engine and myself an incentive to deliver end-to-end quality results.

The fact that this feature does not exist shows that there is something deep within Google's core that is preventing them from addressing SEO spam, just like there is something deep within Airbnb that makes it difficult to filter out Airbnbs with problem reviews.

Google has been coasting for a good long time and now major players are realizing they are wide open for disruption.