I really wish that the Internet Archive would provide bulk access to the Wayback Machine dataset. It would allow for a lot of interesting experimentation and research.
Is that even possible? I don't know the latest size of the IA, but it must be ridiculously huge by now, (1 billion pages a week added) bandwidth cost would be massive. Maybe they could offer a mail-us-a-multi-petabyte-hdd service... Returned a few weeks later full of data :)
Locking the Web Open: A Call for a Distributed Web
41–50 of 100 posts
Re: Locking the Web Open: A Call for a Distributed Web
#42So in my mind the problems that need to be solved are:
Information-Centric Networking > Unstructured Mesh Networking > Distributed Data Storage > P2P Information Retrieval
Re: Locking the Web Open: A Call for a Distributed Web
#43Earlier quoted context omitted.
First...IANAL (I Am Not A Lawyer) You aren't the only one, but with Freenet it's fully encrypted. Let's say you had a Freenet Silk Road application. You won't know it's a Silk Road web page that's being saved along with images of marijuana to your computer unless you go through an indexer/search site and even then you still won't know that those bits of data are stored specifically on your machine. So in order for th…
Not talking about the legal aspects, but, (and I really, really hate to bring up the "think about the children!" argument here) what about if I am unknowingly helping people who create and share child porn? It doesn't matter (to me) if I am on the hook for it or not, I just don't know (ethically?) how I would feel if I knew that was going on via my PC. Drugs I don't give a shit about, and I hate how the "think about…
Re: Locking the Web Open: A Call for a Distributed Web
#44Earlier quoted context omitted.
This may be philosophically correct. But in a non platonic realm you must somehow fund these sorts of things. If it's not possible to make money on them then it may not be feasible to implement for the masses. The keyword being masses there. There all kinds of distributed p2p web projects out there. None of them appeal to a majority though they are all niche products. Most of these proposals forget the need fund mark…
Wikipedia is one of the best sites on the web and didn't think in this way. The whole point of such a system is to remove capital requirements (hardware) from the process of building massive websites. It inherently means less funding is required.
Software development isn't free. Sure, OSS developers can donate their time, but there's a selection process that comes into play.
OSS developers do what they do partly because they enjoy it, and so they tend to gravitate to the types of development that is fun. Fun stuff includes deep systems stuff, algorithms, distributed systems, the flashier aspects of UI/UX, machine learning, etc.
Fixing stupid bugs that stem from stupid compatibility issues and adding stupid features for stupid use cases is not fun. Debugging edge cases that afflict 1% of your users occasionally is not fun. Porting to popular but crap platforms is not fun. Supporting legacy platforms and APIs is not fun, nor is maintaining backward compatibility. Accessibility features and translations are not fun. Supporting right-to-left language is not fun. Rewriting your entire already-working app to support the next web fad (e.g. "responsive mobile design") is not fun.
I could go on forever. There is a really really really long tail of these horrors.
This is why OSS rules in the systems/algorithms/etc. space but drools when you get close to the user. This is why every major end user OS, site, or platform is commercial. People have to be paid to torture themselves with that stuff.
Yet these sorts of problems are precisely the ones that make the difference between something only geeks (with time on their hands) would want to use and something regular people who aren't computer experts would want to use (or computer experts without time on their hands).
For a distributed, decentralized system to challenge the silos of the web, it would absolutely need funding. Eliminating hardware and bandwidth costs is easy; eliminating HR costs is not.
There will never be a volunteer-developed mainstream platform for the same reason there are no volunteer-developed mainstream anything elses. To make a truly polished product of any kind requires pain.
Disclosure: I currently run a distributed networking / SDN startup. The core technology has been up and running for years with few modifications. It started as a side project so I haven't kept careful account, but I'd easily estimate that upwards of 90% of the development time spent on this project has been on trying to get it to the point that it can install and run trouble-free on multiple platforms and is easy enough for mere mortals to use. Getting the core plumbing, crypto, etc. working was the most fun and the most intellectually challenging, but it was a huge minority of time spent. Had there been no chance of a commercial application I would have stopped there and it'd be yet another piece of interesting GitHub networking orphanware, because for the love of God who would voluntarily try to port such a thing to Windows?!? let alone Android (It's in C, so enjoy your JNI pain).
Re: Locking the Web Open: A Call for a Distributed Web
#45I think it's obvious that the current web is decentralized, but is heavily server-based. At the same time, there is something about propagating applications across these servers... Russia can ban Reddit but they can't ban Wordpress. For the moment, that is what we are working on at http://platform.qbix.com (and have been for the past 4 years). Making it easy to have a distributed social network the same way bitcoin m…
> Russia can ban Reddit but they can't ban Wordpress. That's why I'm part of the IndieWeb https://jeena.net/indieweb or http://indiewebcamp.com/ The nicest thing about that all is that I don't need to wait until someone else writes a whole new WWW, my own website already is a small part of the whole big thing, I just make my HTML more machine readable and implement something like pingback (but easier, it is called we…
Could you please get in touch with me by email? You can find it at http://qbix.com/about -- just click on "contact". I would like to find out more about this movement ... I'm beginning to participate more in the Offline First, Distributed Web, Mesh Networking and other such movements. Our company's spent 4 years building a platform that would decentralize social networking, because we see it as the catalyst to giving users control of their own data. Most people in the world are just using centralized services these days, and it's directly related to how difficult it is to make a seamless social layer for the web. So I think that we're solving a solution parallel to what bitcoin did with money. A good solution unleashes new possibilities, like the Web itself did, like Email did.
Anyway, reach out if you can! - Greg
Re: Locking the Web Open: A Call for a Distributed Web
#46Re: Locking the Web Open: A Call for a Distributed Web
#47Earlier quoted context omitted.
First...IANAL (I Am Not A Lawyer) You aren't the only one, but with Freenet it's fully encrypted. Let's say you had a Freenet Silk Road application. You won't know it's a Silk Road web page that's being saved along with images of marijuana to your computer unless you go through an indexer/search site and even then you still won't know that those bits of data are stored specifically on your machine. So in order for th…
Not talking about the legal aspects, but, (and I really, really hate to bring up the "think about the children!" argument here) what about if I am unknowingly helping people who create and share child porn? It doesn't matter (to me) if I am on the hook for it or not, I just don't know (ethically?) how I would feel if I knew that was going on via my PC. Drugs I don't give a shit about, and I hate how the "think about…
Re: Locking the Web Open: A Call for a Distributed Web
#48There's the problem that distributing content means someone else pays for storing and serving it. This is part of what killed USENET, once the binary groups (mostly pirated stuff and porn) became huge. There's a scaling problem with replication.
Federated networks are interesting, and there are several federated social networks. A few even have a number of servers in two digits. You could have a federated Facebook replacement that costs each user under a dollar a month at current hosting prices. No ads. The concept is not getting any traction.
Kahle wants a system with "easy mechanisms for readers to pay writers." That's either micropayments or an app store, both of which are worse than the current Web.
Re: Locking the Web Open: A Call for a Distributed Web
#49I've participated in some file-sharing litigation which has made it very clear to me that decentralized P2P systems are not inherently more anonymous than other technologies. In fact, there's a cottage industry of P2P monitoring companies that participate as peers in the P2P networks and record detailed information about the IP addresses of peers that uploaded and downloaded particular files. There are often paradoxes where decentralization helps privacy and anonymity in some ways but harms it in others -- for example, if you run your own mail server instead of using Gmail, then you've prevented Google from knowing who communicates with whom, but allowed a network adversary to learn that information directly, where the network adversary might not know the messaging relationships if everyone on the network used Gmail.
I guess a related point is that information about who is doing what online exists somewhere by default, unless careful privacy engineering reduces the amount of information that's out there. Making the simplest kinds of architectural changes could just shift the location where the information exists, for example from Google or Yahoo or Amazon to dozens of random strangers, some of whom might be working for an adversary.
Re: Locking the Web Open: A Call for a Distributed Web
#50I'm happy to see this article, and it reminds me of things that others have been talking about for some time (for example, the "Redecentralize" community). I've participated in some file-sharing litigation which has made it very clear to me that decentralized P2P systems are not inherently more anonymous than other technologies. In fact, there's a cottage industry of P2P monitoring companies that participate as peers…
Anything less than that is like using snake oil crypto: it might make you feel good, but it's not really there.