Decentralize All the Things

I've been hoping we return to dencentralization for awhile http://timepedia.blogspot.com/2008/05/decentralizing-web.htm... but it seems the economics, bandwidth, and technology currently doesn't favor it.

Consider what it would take to do decentralized search with a reasonable latency. Everyone would have to have a copy of Google's index, or at atleast a reasonable chunk of it, or proximity to some mesh node with a reasonable chunk of it.

But mobile devices and mobile bandwidth are not really efficient as nodes to have complete copies, or partial copies. They're low on memory, and it's not really power efficient or network efficient for them to do the work.

But what's worse is the problem is updating the index. If everyone ran a distributed crawler, there'd have to be some way to coordinate who crawls what and make sure there is not too much duplication of effort (else we waste network bandwidth and have a tragedy of the commons) You also run into problems of people trying to (and economically incentivized to) poison the index, and so you'd need an overhead of duplicate just to avoid that.

It's not impossible to arrange a distributed crawl/map/reduce over millions of clients, the Folding@Home and other crowdsourced computation platforms show it can be done. But they don't have to manage replication and updating of petabyte scale databases, the problems they coordinate are easily sliced up and the results to be transmitted back aren't large.

I think this tends to drive the network back towards super-nodes, as the leaf nodes aren't efficient, and the cost associated with doing the work can be pushed onto others (free rider problem). Eventually, the people running the super-nodes tend to become relied upon by everyone, and those become subject to the same kinds of problems we see today -- government attacks, hackers, privacy invasion, etc.

It's not like SMTP/USENET weren't once relatively decentralized networks. But as millions and then billions of people came on line, the cost to run SMTP or USENET nodes was too high in complexity for most people to run their own. Small and medium dialup ISPs arose to handle that need. But since these were commodities, and little differentiation other than price, the prices were quickly driven to the marginal cost, and there isn't much money in running an email server for a few hundred people, you need scale. That eventually drove consolidation, and that's why today, you either get your email from Gmail, Yahoo, AOL, iCloud, Hotmail, or one of the other larger players.

It may very well be that for some types of problems, there is inherent efficiency in centralization. We should look at perhaps how to change the way the services work to favor cost effectiveness of decentralization, so that rather than it being favored for ideological or aesthetic reasons, it is favored by an economic gradient.

While I assume one could, I'm not sure it would be easy to implement web search in Ethereum?

First of all, you would need a way for a contract to be able to access webpages to get information about links. I the contract could pay people to give it the content of a webpage? I'm not sure how it would verify it for non https websites, without getting copies from multiple users, which would cause problems if the webpage was not static.

Second of all, the memory to store what pages link to what pages would, I think, be expensive for contract storage. Maybe if it used swarm?

I guess it would use swarm.

Still, the contract would have to have a short running time (in order to be relatively inexpensive). Maybe if instead of the contract producing the results, it would have others create the results, and then verify them in some way, and reward those that created the results?

I don't know.

I don't really know my stuff, but it seems to me that implementing a decentralized search engine with Ethereum would be fairly difficult.

The elephant in the room that nobody seems to see is that most ISP upload speeds are a quarter at best of the download. I have a 50megabit service with 5megabit upload. When we talk about internet freedom/net neutrality, its always talk regarding big companies being clocked, should Netflix be-able to use 30% of the nations internet resources without paying more...but wait consumers are paying for the connections.. Its a pricing debate. This occurrence of decentralization is the result of the restriction of possibility,we are stuck in the most part with the options that the isp's allow. Which are structured to get more money out of servers of the internet by restricting the availability of high or even equal uploads. even with my own home server I can't host a website that allows for any reasonable dynamic exchange.

Back in the day, there was Gilmour who would publish an article about Twitter every week on TC. "Twitter this", "twitter that", "twitter is the answer to any possible problem out there". Now it's blockchains. That's TC's MO, nothing new.

Yeah, I thought these string indexing algorithms are inherently decentralized. Ie. the data structures themselves are big trees.