Live data from Hacker News

Locking the Web Open: A Call for a Distributed Web

brewster.kahle.org

41–50 of 100 posts

Re: Locking the Web Open: A Call for a Distributed Web

#41
post #39

I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset. It would allow for a lot of interesting experimentation and research.

Is that even possible? I don't know the latest size of the IA, but it must be ridiculously huge by now, (1 billion pages a week added) bandwidth cost would be massive. Maybe they could offer a mail-us-a-multi-petabyte-hdd service... Returned a few weeks later full of data :)

It's totally possible, they already have the infrastructure in place and 14PB of data available for download. Unfortunately the Wayback Machine data is not currently exposed publicly.

Re: Locking the Web Open: A Call for a Distributed Web

#42
I agree with one of the comments on that page that decentralizing the Internet is fundamental to decentralizing the Web.

So in my mind the problems that need to be solved are:

Information-Centric Networking > Unstructured Mesh Networking > Distributed Data Storage > P2P Information Retrieval

Re: Locking the Web Open: A Call for a Distributed Web

#43
post #37
post #31

Earlier quoted context omitted.

First...IANAL (I Am Not A Lawyer) You aren't the only one, but with Freenet it's fully encrypted. Let's say you had a Freenet Silk Road application. You won't know it's a Silk Road web page that's being saved along with images of marijuana to your computer unless you go through an indexer/search site and even then you still won't know that those bits of data are stored specifically on your machine. So in order for th…

Not talking about the legal aspects, but, (and I really, really hate to bring up the "think about the children!" argument here) what about if I am unknowingly helping people who create and share child porn? It doesn't matter (to me) if I am on the hook for it or not, I just don't know (ethically?) how I would feel if I knew that was going on via my PC. Drugs I don't give a shit about, and I hate how the "think about…

With a fully distributed model, everyone is essentially running a backbone server. If you don't feel comfortable with such an arrangement, then you'd probably have to opt out. There are plenty of people willing to put up with it, evidenced by the number of companies who have no problem operating the current Internet backbone despite knowing for a fact that their networks are used to distribute child porn and other illegal things. Generally, I think the law is on the side of the distributor, but that's only a legal consolation, not a moral one. I just don't see a way around it. To be fair though, the likelihood of this happening is probably going to be much lower than for a backbone provider, especially if users only serve up content they've consumed themselves (seems like a logical assumption in the distributed system I'm thinking about, but does not have to hold true for all such systems).

Re: Locking the Web Open: A Call for a Distributed Web

#44
post #19

Earlier quoted context omitted.

This may be philosophically correct. But in a non platonic realm you must somehow fund these sorts of things. If it's not possible to make money on them then it may not be feasible to implement for the masses. The keyword being masses there. There all kinds of distributed p2p web projects out there. None of them appeal to a majority though they are all niche products. Most of these proposals forget the need fund mark…

Wikipedia is one of the best sites on the web and didn't think in this way. The whole point of such a system is to remove capital requirements (hardware) from the process of building massive websites. It inherently means less funding is required.

Hardware is only a tiny, tiny part of the cost of developing a system for general use by huge numbers of people. If you don't believe me, do a price check on cloud hosting and then keep in mind that cloud hosting must be profitable so the prices you're seeing have some non-trivial margin attached.

Software development isn't free. Sure, OSS developers can donate their time, but there's a selection process that comes into play.

OSS developers do what they do partly because they enjoy it, and so they tend to gravitate to the types of development that is fun. Fun stuff includes deep systems stuff, algorithms, distributed systems, the flashier aspects of UI/UX, machine learning, etc.

Fixing stupid bugs that stem from stupid compatibility issues and adding stupid features for stupid use cases is not fun. Debugging edge cases that afflict 1% of your users occasionally is not fun. Porting to popular but crap platforms is not fun. Supporting legacy platforms and APIs is not fun, nor is maintaining backward compatibility. Accessibility features and translations are not fun. Supporting right-to-left language is not fun. Rewriting your entire already-working app to support the next web fad (e.g. "responsive mobile design") is not fun.

I could go on forever. There is a really really really long tail of these horrors.

This is why OSS rules in the systems/algorithms/etc. space but drools when you get close to the user. This is why every major end user OS, site, or platform is commercial. People have to be paid to torture themselves with that stuff.

Yet these sorts of problems are precisely the ones that make the difference between something only geeks (with time on their hands) would want to use and something regular people who aren't computer experts would want to use (or computer experts without time on their hands).

For a distributed, decentralized system to challenge the silos of the web, it would absolutely need funding. Eliminating hardware and bandwidth costs is easy; eliminating HR costs is not.

There will never be a volunteer-developed mainstream platform for the same reason there are no volunteer-developed mainstream anything elses. To make a truly polished product of any kind requires pain.

Disclosure: I currently run a distributed networking / SDN startup. The core technology has been up and running for years with few modifications. It started as a side project so I haven't kept careful account, but I'd easily estimate that upwards of 90% of the development time spent on this project has been on trying to get it to the point that it can install and run trouble-free on multiple platforms and is easy enough for mere mortals to use. Getting the core plumbing, crypto, etc. working was the most fun and the most intellectually challenging, but it was a huge minority of time spent. Had there been no chance of a commercial application I would have stopped there and it'd be yet another piece of interesting GitHub networking orphanware, because for the love of God who would voluntarily try to port such a thing to Windows?!? let alone Android (It's in C, so enjoy your JNI pain).

Re: Locking the Web Open: A Call for a Distributed Web

#45
post #14
post #9

I think it's obvious that the current web is decentralized, but is heavily server-based. At the same time, there is something about propagating applications across these servers... Russia can ban Reddit but they can't ban Wordpress. For the moment, that is what we are working on at http://platform.qbix.com (and have been for the past 4 years). Making it easy to have a distributed social network the same way bitcoin m…

> Russia can ban Reddit but they can't ban Wordpress. That's why I'm part of the IndieWeb https://jeena.net/indieweb or http://indiewebcamp.com/ The nicest thing about that all is that I don't need to wait until someone else writes a whole new WWW, my own website already is a small part of the whole big thing, I just make my HTML more machine readable and implement something like pingback (but easier, it is called we…

IndieWeb looks great. I am going to try to get involved with it.

Could you please get in touch with me by email? You can find it at http://qbix.com/about -- just click on "contact". I would like to find out more about this movement ... I'm beginning to participate more in the Offline First, Distributed Web, Mesh Networking and other such movements. Our company's spent 4 years building a platform that would decentralize social networking, because we see it as the catalyst to giving users control of their own data. Most people in the world are just using centralized services these days, and it's directly related to how difficult it is to make a seamless social layer for the web. So I think that we're solving a solution parallel to what bitcoin did with money. A good solution unleashes new possibilities, like the Web itself did, like Email did.

Anyway, reach out if you can! - Greg

Re: Locking the Web Open: A Call for a Distributed Web

#47
post #37
post #31

Earlier quoted context omitted.

First...IANAL (I Am Not A Lawyer) You aren't the only one, but with Freenet it's fully encrypted. Let's say you had a Freenet Silk Road application. You won't know it's a Silk Road web page that's being saved along with images of marijuana to your computer unless you go through an indexer/search site and even then you still won't know that those bits of data are stored specifically on your machine. So in order for th…

Not talking about the legal aspects, but, (and I really, really hate to bring up the "think about the children!" argument here) what about if I am unknowingly helping people who create and share child porn? It doesn't matter (to me) if I am on the hook for it or not, I just don't know (ethically?) how I would feel if I knew that was going on via my PC. Drugs I don't give a shit about, and I hate how the "think about…

Maybe think of it this way, if you were a construction worker and helped build an interstate, you'd be helping all sorts of criminals do all sorts of horrible things. Plus innocent people would die in car crashes on the road you helped to build. But we can agree that the utility of an interstate far outweighs these drawbacks.

Re: Locking the Web Open: A Call for a Distributed Web

#48
Kahle's approach works only for static content. It's not hard to distribute static content; BitTorrent does it just fine. The Internet Archive stores static content. Kahle thinks in terms of static content, because that's what the Internet Archive does. But it's less of the Web today. Despite that, it's good to have a way to distribute static content. Academic publishing, after all, is almost all static content. That should be widely distributed. It's not like academic journals pay their authors.

There's the problem that distributing content means someone else pays for storing and serving it. This is part of what killed USENET, once the binary groups (mostly pirated stuff and porn) became huge. There's a scaling problem with replication.

Federated networks are interesting, and there are several federated social networks. A few even have a number of servers in two digits. You could have a federated Facebook replacement that costs each user under a dollar a month at current hosting prices. No ads. The concept is not getting any traction.

Kahle wants a system with "easy mechanisms for readers to pay writers." That's either micropayments or an app store, both of which are worse than the current Web.

Re: Locking the Web Open: A Call for a Distributed Web

#49
I'm happy to see this article, and it reminds me of things that others have been talking about for some time (for example, the "Redecentralize" community).

I've participated in some file-sharing litigation which has made it very clear to me that decentralized P2P systems are not inherently more anonymous than other technologies. In fact, there's a cottage industry of P2P monitoring companies that participate as peers in the P2P networks and record detailed information about the IP addresses of peers that uploaded and downloaded particular files. There are often paradoxes where decentralization helps privacy and anonymity in some ways but harms it in others -- for example, if you run your own mail server instead of using Gmail, then you've prevented Google from knowing who communicates with whom, but allowed a network adversary to learn that information directly, where the network adversary might not know the messaging relationships if everyone on the network used Gmail.

I guess a related point is that information about who is doing what online exists somewhere by default, unless careful privacy engineering reduces the amount of information that's out there. Making the simplest kinds of architectural changes could just shift the location where the information exists, for example from Google or Yahoo or Amazon to dozens of random strangers, some of whom might be working for an adversary.

Re: Locking the Web Open: A Call for a Distributed Web

#50
post #49

I'm happy to see this article, and it reminds me of things that others have been talking about for some time (for example, the "Redecentralize" community). I've participated in some file-sharing litigation which has made it very clear to me that decentralized P2P systems are not inherently more anonymous than other technologies. In fact, there's a cottage industry of P2P monitoring companies that participate as peers…

The only mechanism I'm aware of that truly allows anonymity over your own connection (or a connection that can be tied to you) is onion routing. On top of that, you must do it from a separate device or isolated VM to prevent hardware fingerprinting.

Anything less than that is like using snake oil crypto: it might make you feel good, but it's not really there.

Post reply on HN