Hyperdrive v10 – a peer-to-peer filesystem • Hacker News Board

Re: Hyperdrive v10 – a peer-to-peer filesystem

#121

Post by Reelin » Sat, May 16, 2020, 2:54 AM UTC

dpflug wrote: post #119

Reelin wrote: post #94

Earlier quoted context omitted.

Perhaps I'm completely misunderstanding something but don't centralized self hosted cloud-plus-app services address this exact usecase? Things like NextCloud, Seafile, and Sandstorm? The only major drawback seems to be that you have to host the physical hardware yourself due to lack of solid end-to-end encryption for most platforms. Sandstorm might have it (I'm not clear on what's client- and what's server- side ther…

SyncThing does support end-to-end encryption, and has for a while: https://docs.syncthing.net/users/security.html#sync-connecti...

By end-to-end, I meant that the server can't see your data. For example, the idea of deploying one or more SyncThing instances to cloud providers that would never see your unencrypted data. There's an open GitHub issue about it since 2014 (https://github.com/syncthing/syncthing/issues/109).

Re: Hyperdrive v10 – a peer-to-peer filesystem

#122

Post by sktrdie » Sat, May 16, 2020, 10:19 AM UTC

nobodywillobsrv wrote: post #103

sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

Would sqltorrent setup make sense for sharing scraped/pulled data amongst users. So each user can run the data-extraction themselves or check if anyone has ingress chunks to their liking on the swarm? Everying is append-only content addressable at it's base. I've been looking around IPFS, dat, hyperdrive etc and it seems like dat is the most natural setting for this but sqltorrent is new to me.

"Sharing data amongst user" - torrents excel at this. Do you have a specific use-case in mind?

Re: Hyperdrive v10 – a peer-to-peer filesystem

#123

Post by namibj » Sun, May 17, 2020, 2:34 AM UTC

black_puppydog wrote: post #114

namibj wrote: post #110

Earlier quoted context omitted.

Actually, there is a solution against this. Just combine https://www.bittorrent.org/beps/bep_0030.html (Merkle-tree-based hashing) with https://www.bittorrent.org/beps/bep_0039.html (Feed-URL based updates), and in some settings also https://www.bittorrent.org/beps/bep_0047.html (Specifically the padding files, so that flat files inside a torrent can also be efficiently shared in arbitrary combinations of non-partial…

All those BEPs are in "Draft" status. Okay, libtorrent implements two of them. But also, BEP 39 (Updating Torrents Via Feed URL) doesn't really fit very well into the fully distributed setting because of the centralized URL part. So now to update the torrent file you need a mechanism for having a mutable document you can update in a distributed but signed way. Or you could make an append only feed of sequential torre…

The distributed-but-signed way is there in https://www.bittorrent.org/beps/bep_0046.html (Updating Torrents Via DHT Mutable Items).

That draft status is of little practical relevance, though, if nothing changed for years, and no one voiced well-founded critic on the technical details.

I do agree though that Hyperdrive is different from what the bittorrent ecosystem has to offer. I too like not reinventing the wheel where that's not necessary, as you recommend there. I'll leave you the list of BEPs for further reading, in case you're interested: https://www.bittorrent.org/beps/bep_0000.html

Re: Hyperdrive v10 – a peer-to-peer filesystem

#124

Post by black_puppydog » Mon, May 18, 2020, 11:43 AM UTC

namibj wrote: post #123

black_puppydog wrote: post #114

Earlier quoted context omitted.

All those BEPs are in "Draft" status. Okay, libtorrent implements two of them. But also, BEP 39 (Updating Torrents Via Feed URL) doesn't really fit very well into the fully distributed setting because of the centralized URL part. So now to update the torrent file you need a mechanism for having a mutable document you can update in a distributed but signed way. Or you could make an append only feed of sequential torre…

The distributed-but-signed way is there in https://www.bittorrent.org/beps/bep_0046.html (Updating Torrents Via DHT Mutable Items). That draft status is of little practical relevance, though, if nothing changed for years, and no one voiced well-founded critic on the technical details. I do agree though that Hyperdrive is different from what the bittorrent ecosystem has to offer. I too like not reinventing the wheel w…

I've been keeping an eye on that list for a long time. There's some really cool stuff in there, and I think bittorrent has really been within reach of being "simply good enough for most applications" for quite some time now. And the massive user base is of course a good thing there, especially if you're talking more about archival projects.