The Destruction of the Web

Destruction just seems like such a harsh word.

Any time you measure something, you impact what you measure. Without being pedantic about the Physics, look around. If you measure when people get in and our of work, they will optimize around it, and other things may suffer. If you measure defects in code, people will optimize it.

But back to the web... Search by definition is a measurement or prediction of usefulness. It will definitely impact what gets searched. But that collateral damage can be minimized. And it some cases it can improve the web experience, though I wouldn't bank on it. I certainly don't view the Web as a pristine wilderness being destroyed.

My concern -- and this is a total layman's observation -- is that if someone comes up with a good way to "spam" the academic research circuit, will we be able to tell? Most of us remember the recent Reinhart-Rogoff incident in which a massively flawed paper (think, Excel-based) wasn't challenged until a curious grad student took notice: http://phys.org/news/2013-04-excel-austerity-economics-paper...

There are other issues in all of that, but for the current discussion, I think it's enough to argue that discovery of exaggeration (the equivalent of resume padding) is still a very difficult problem in academic papers and citations. Do researchers have the tools to methodically sift out the good from the meh? It doesn't seem so. Combine that with the lack of incentive (as seems to be the case in Reinhart-Rogoff) to disprove published findings, and you have a scenario in which gaming the system seems quite doable.

At the cost of result quality, one could exploit social networks as a possible solution: promote the pages that your neighbors visit for an extended period of time (and thus deem to be of high value) by a factor corresponding to said neighbor's distance from you. If the queries are slightly randomized such that their order differs per person and per session, then new possible results could tested.

"Destruction of the Web" is a bit hyperbolic. I tried the SEO game for the interest of my clients, I've come to realize that it's a better long term strategy to simply focus on good, relevant content and let nature take it's course. When you're publishing content targeted at a certain audience, spammy visits to your website just increase your bandwidth cost without much benefit.

Granted there is inherent benefit in coming in on top of a google search, but time and time again, I've seen good content naturally dominate and stay on top. Occasionally some black had spammer comes out on top.

And truth of the matter is, I think it's google/'s job to figure out a way to discern good content from spam.

Google Dominated early on, b/c they were able to parse thru a lot of the garbage and find you what you were looking for. There will always be spammers. And if this whole arms race thing is true, I think the spammers are likely to hit a ceiling before Google or another search engine is.

As a programmer, my initial instinct was to write code to counter what Google's algorithm would expect. Then, I thought, if I'm going to put in that much work, why the hell not build a better search engine myself?

I believe that if Google doesn't do a great job of getting the best possible results, not only do they face a treat from other giants like bing, but also from crafty programmers who may be writing black hat seo crap now and have the epiphany to try to build a better search engine (yes... I know... I think PG has a real point with the search engine thing)

Gad, I am so sick of SEO. Not Jacques article, but SEO in general.

Google owns the game. They run the game on a computer. Ergo, if you want people naturally coming from Google, you must do things its computer likes.

Only they won't tell you that. Instead, they'll offer platitudes like "write good content and the users will come" when we all know you could write great content until the cows come home and if nobody links to you, you ain't getting no traffic.

And I think it's unfair to call all these guys leeches, miscreants, or whatever. I don't like a lot of the things they do, but I also respect the fact that I live in a first world country. I have a good way of living. If I were terribly impoverished and only had to spam a lot to feed my family? I'd do it. We assume everybody else on the web lives the same lives that we do. We also are getting this quasi-religious thing going on where Google must return what I want at the top of the search results. If it does not, somebody has sinned. I'm not drinking that cool-aid.

I'm with Jacques on the solution: a new protocol and the elimination of single-points-of-failure. This thing where Google keeps updating it's algorithm and tens of thousands of people keep gaming the system has to stop. It's not healthy behavior either for Google or for the spammers. And it's destroying the web.

Sidebar: you know, if you think about it, with all the walled gardens and vendors refusing common protocols and such, the web itself is under attack from multiple angles.