HTTP is obsolete – it's time for the distributed, permanent web (2015)

What was Apple's market share of desktops at the time? And whether you were talking people through a Trumpet WinsSock install or I was doing it in person, it wasn't going to get done that way, just too slow. Sure lots of physical infrastructure (modems etc...) had to be added, no disagreement there. Anyway, it's all a long time ago now. A funny aside, thinking of Apple in those days reminded me of "Cyberdog" - https://en.wikipedia.org/wiki/Cyberdog. I wonder if anybody has done a side-by-side of Cyberdog vs Safari. Things change.

I realize you plan to do a Show HN but in case I don't see it, could you answer a few quick questions:

1. How is this meaningfully different to WebDAV?

2. Is the assumption that web apps export stuff to your pod from time to time, or actually use it as the primary storage? If the former, isn't it more or less the same idea as Google Takeout, if the latter how do apps handle the possibility of slow pods, outages or the need to use relational databases for storage? When building server side apps you do normally need tight control over storage.

> you still need to handle dynamic behaviour and that's most of the work.

I'm not sure what you think would be difficult exactly. You've said that archive.org has already done the programming needed to ensure dynamic resources are discovered, and now those resources are content ids rather than URLs. Nothing's really changed on this point.

> Relying on volunteers means that you need far more copies because nobody has a commitment to provide resources or even tell you if they decide to stop

Yes, but you would also have many more volunteers. Many people who wouldn't donate financially would donate CPU and storage. We saw this with SETI@home and folding@home, for instance.

> nobody has a commitment to provide resources or even tell you if they decide to stop

Why not? If you provide a client to participate as a storage node for archive.org, like SETI@home, then they would know your online/offline status and how much storage you're willing to donate. If you increase/decrease the quota, it could notify the network of this change.

> 1. How is this meaningfully different to WebDAV?

Webpods is more like git than WebDAV. It allows apps/users to store data in logs, and the data can be records (strings) or files. If bob is syncing from alice, he'd pull all entries from the commit-id until which he has previously synced.

An app will store data in a pod (which has a unique hostname) such as instagram.jeswin.someprovider.com. Each pod can have multiple logs, such as "friends", "albums", "comments" etc. Logs have permissions attached to them, which control who can read those logs. There are similarities to WebDAV here, but again it's more like how we use git.

> Is the assumption that web apps export stuff to your pod from time to time, or actually use it as the primary storage? If the former, isn't it more or less the same idea as Google Takeout, if the latter how do apps handle the possibility of slow pods, outages or the need to use relational databases for storage? When building server side apps you do normally need tight control over storage.

Apps are expected to be local-first, though they aren't forced to. You'd write to the local database, and simultaneously sync with the pod. Similarly, if you're pulling data from friends, those would (most likely) be stored locally as well.

Slow pods are a problem, but I hope people would generally prefer reliable pod service providers. In the same way Dropbox gives you some guarantees of reliability. If the app is designed to be local-first, the user is not immediately prevented from using the app while the network is down; and syncing can happen once connectivity is regained.

Relational databases and schemas are not supported on Pods, it's just an immutable log. Most apps should do event-sourcing (https://martinfowler.com/eaaDev/EventSourcing.html), wherein they write to an event log. But this stream (of events) could be processed into an more easily queryable view.

Of course, this won't work for all kinds of apps. It works well for apps handling personal data or for collaboration tools; such as slack, project management tools, instagram, google photos, music collections etc. On the other hand, it's not a good fit for apps in which the data needs to be centralized. Such as ecommerce, banking, insurance, delivery services etc.

> IA actively maintains the code which does this and any competing project would need to spend the same time on that for the same reasons.

I'm not sure why they would have to compete. They're literally solving the same problem in basically the same way. I see no reason to fork this code.

> That's an interesting theory but do we have any evidence suggesting that it's likely? In particular, SETI@home / folding@home did not involve either substantial resource commitments or potential legal problems, both of which would be a concern for a web archiving project.

But this isn't a concern of a web archiving project any more if content-based addressing becomes the standard, because pervasive caching is built into the protocol itself. Publishing anything on such a network means you are already giving up some control you would otherwise have in where this content will be served from, how it's cached, how long it lasts, etc.

> All of what you're talking about is voluntary. One challenge of systems like this is that you don't know whether a node which simply disappears is going to come back or you need create a new replica somewhere else.

Yes, you would have to be more pessimistic and plan for more redundancy than you would otherwise need. Each node in a Google-scale distributed system has a low expected failure rate, but they still see regular failures. No doubt they have a minimum redundancy calculation based on this failure rate. The same logic applies here, but the failure rate would likely have to be jacked up.

> Since you don't have an SLA, you have to take conservative approach — lots of copies, geographically separated, etc. — which reduces the total system capacity and introduces performance considerations.

Whether there would be performance problems isn't clear. Content-based addressing is already fairly slow (at this time), but once content is resolved, fragments of content can be delivered from multiple sources concurrently, and from more spatially close sources. Higher latency, but more parallelism.

I'm not willing to invest the time needed to gather the data you're asking about to actually quantify all of the requirements, but despite the points you've raised, I still don't see any real obstacles in principle.