Would Hyperswarm be useful for setting up mesh-like Wireguard networks automatically? We need some solution for this ASAP - IPv6 not coming, CGNATs everywhere.
That'd be zerotier or tailscale
Hyperdrive v10 – a peer-to-peer filesystem
111–120 of 124 posts
Re: Hyperdrive v10 – a peer-to-peer filesystem
#112Can you limit the synced size on clients (like an LRU cache) ?
Re: Hyperdrive v10 – a peer-to-peer filesystem
#113Great work. Planning to read about it. I, and I am sure others, are continually intrigued about this space and I hold out hope that the definitive solution appears soon (along with the solution to micropayments !!).
On a slightly related note, is anyone interested in having a discussion about how to layer on top of all these "drive" systems a `HDFS` like drive that would have n block replication across different sources, and trying to interpret any given source (dropbox, hyperdrive, google drive, etc..) blocks to make sense of what is being stored there would render the person confused?
:)
Re: Hyperdrive v10 – a peer-to-peer filesystem
#114Earlier quoted context omitted.
Last I checked you can't update a torrent. So if Wikipedia changes even a single letter, you'd need to download all the data once more
Actually, there is a solution against this. Just combine https://www.bittorrent.org/beps/bep_0030.html (Merkle-tree-based hashing) with https://www.bittorrent.org/beps/bep_0039.html (Feed-URL based updates), and in some settings also https://www.bittorrent.org/beps/bep_0047.html (Specifically the padding files, so that flat files inside a torrent can also be efficiently shared in arbitrary combinations of non-partial…
All those BEPs are in "Draft" status. Okay, libtorrent implements two of them. But also, BEP 39 (Updating Torrents Via Feed URL) doesn't really fit very well into the fully distributed setting because of the centralized URL part.
So now to update the torrent file you need a mechanism for having a mutable document you can update in a distributed but signed way. Or you could make an append only feed of sequential torrent urls... oh wait.
My point is: Hyperdrive's scope is sufficiently different from your proposed solution that yes, you could probably rely on existing tools (and I have much love for bittorrent based solutions!) but it starts feeling like shoehorning the problem into a solution that doesn't quite fit.
Re: Hyperdrive v10 – a peer-to-peer filesystem
#115Earlier quoted context omitted.
It seems like the blog post answers your question pretty thoroughly. The Hyperdrive index and the protocol are tuned for this use case, making it scale to being able to host a Wikipedia clone. BitTorrent FS + SQlite are not tuned for this use case.
Wikipedia’s text history absolutely fits on a tiny hard drive and is easy to get a replica of.
Compressed with 7-Zip, sure, but uncompressed, the entire thing takes up 10TB. The Hyperdrive post doesn't mention compression at all, so the comparison should be without it.
> As of June 2015, the dump of all pages with complete edit history in XML format at enwiki dump progress on 20150602 is about 100 GB compressed using 7-Zip, and 10 TB uncompressed.
From: https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia#Si...
It's a bit of a nitpick either way because you're right, Wikipedia may not be the best example because 10TB is still relatively small.
Re: Hyperdrive v10 – a peer-to-peer filesystem
#116Great work. Planning to read about it. I, and I am sure others, are continually intrigued about this space and I hold out hope that the definitive solution appears soon (along with the solution to micropayments !!). On a slightly related note, is anyone interested in having a discussion about how to layer on top of all these "drive" systems a `HDFS` like drive that would have n block replication across different sour…
One more question... How effective would this be for a `network questionable` environment, if I were let's say doing a mobile app and wanted to incorporate this type of solution in for an offline first type of experience?
Re: Hyperdrive v10 – a peer-to-peer filesystem
#117Great work. Planning to read about it. I, and I am sure others, are continually intrigued about this space and I hold out hope that the definitive solution appears soon (along with the solution to micropayments !!). On a slightly related note, is anyone interested in having a discussion about how to layer on top of all these "drive" systems a `HDFS` like drive that would have n block replication across different sour…
One more question... How effective would this be for a `network questionable` environment, if I were let's say doing a mobile app and wanted to incorporate this type of solution in for an offline first type of experience?
Hyperdrive (and the stack it's built on more generally) works great offline. All the drives you own are still editable without the network, and changes will sync when you come online.
As for 'network questionable' and mobile, here are a few of the other projects building on Hypercore that have made those a priority [0] [1] [2].
The Hyperdrive daemon as it's currently built wouldn't fare too well in a bandwidth and/or battery constrained environment (wasn't designed for that), but a mobile solution is definitely on our radar.
Re: Hyperdrive v10 – a peer-to-peer filesystem
#118Earlier quoted context omitted.
What do you mean? The author (say wikipedia owners) can change the db as they usually would change (using UPDATE queries say). Those write queries will result in the least-amount of disk-pages updates. In the torrent world this equals a minimum set of pieces modified and needed to be downloaded by users.
Last I checked you can't update a torrent. So if Wikipedia changes even a single letter, you'd need to download all the data once more
No, the pieces you downloaded can be reused for the new torrent download. The pieces will effectively have the same hash hence can be reused for the new digest: http://bittorrent.org/beps/bep_0038.html
This is also why sqlite is a good choice because it's highly optimized to do the least amount of changes to its "pieces" when an update occurs.
If you're implementing this behavior, trying to manage all kinds of different queries, building a querying engine on top of that, optimizing for efficiency and reliability, you're effectively rewriting a database. Sure you can do it, but why not take advantage of battle-tested off-the-shelf stuff for things like "databases" (sqlite) and/or "distributing data" (torrent)?
Re: Hyperdrive v10 – a peer-to-peer filesystem
#119Earlier quoted context omitted.
Do you have any good solutions for this exact use case yet?
Perhaps I'm completely misunderstanding something but don't centralized self hosted cloud-plus-app services address this exact usecase? Things like NextCloud, Seafile, and Sandstorm? The only major drawback seems to be that you have to host the physical hardware yourself due to lack of solid end-to-end encryption for most platforms. Sandstorm might have it (I'm not clear on what's client- and what's server- side ther…
SyncThing does support end-to-end encryption, and has for a while: https://docs.syncthing.net/users/security.html#sync-connecti...
Re: Hyperdrive v10 – a peer-to-peer filesystem
#120Been following this for years, congratulations on your release. I looked at dat for building P2P services, but found that ZeroNet was actually more capable for "real" services - maybe now is the time to reevaluate that, especially given the improvements to large (both deep and wide) archives and the new hole punching. Can you please mention your thoughts on: - Discoverability of content in Hyperswarm (DHT Search/"Sup…
Check out the dependency hyper-protocol, specifically the extension messaging, custom-messaging has been around for a while allowing you to define your own protocol-extensions and peer behaviour. hyper-protocol-v7 came with an option for doing authenticated peer handshakes as well.