Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperdrive v10 – a peer-to-peer filesystem

blog.hypercore-protocol.org

Post reply on HN
Print view View original

101–110 of 124 posts

Re: Hyperdrive v10 – a peer-to-peer filesystem

#101

Post by sktrdie »

fenwick67 wrote: post #84
sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

The big asterisk is, this only works if your database never changes.

What do you mean? The author (say wikipedia owners) can change the db as they usually would change (using UPDATE queries say). Those write queries will result in the least-amount of disk-pages updates. In the torrent world this equals a minimum set of pieces modified and needed to be downloaded by users.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#102

Post by black_puppydog »

gnu wrote: post #98
jhardy54 wrote: post #87

Earlier quoted context omitted.

That's very interesting, thanks for the links. I'm working on Scuttlebutt (like Dat/Hypercore) and have been working on reimplementing our stack with 'boring' tooling like SQLite and HTTP, and I've been really enjoying it so far. I'm going to read your blog post now, thanks a lot for the new info.

Interesting! Is there any document that I can read about your reimplementation? Or any code?

Christian has also put some work into the underlying database and such lately, but the user facing part of that is Oasis [1] which aims to be an ssb interface that has a no-JS UI, with all the logic being handled by the (locally running) nodeJS server.

[1]: https://github.com/fraction/oasis

Re: Hyperdrive v10 – a peer-to-peer filesystem

#103

Post by nobodywillobsrv »

sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

Would sqltorrent setup make sense for sharing scraped/pulled data amongst users. So each user can run the data-extraction themselves or check if anyone has ingress chunks to their liking on the swarm? Everying is append-only content addressable at it's base.

I've been looking around IPFS, dat, hyperdrive etc and it seems like dat is the most natural setting for this but sqltorrent is new to me.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#104

Post by czei002 »

Sharing an append only log can be quite treacherous for users that are unaware of it. For example, when accidentally including some confidential stuff. A not so security aware person may think a quick delete fixes it and depending on the situation this might event be true, but its still in the log... Not familiar with Hyperdrive so please somebody correct me if I am wrong and this case is handled.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#105

Post by thijsvandien »

hinkley wrote: post #97
mceachen wrote: post #90

Earlier quoted context omitted.

Synology hardware is pretty close to what you're describing. Light on front goes from green to orange and you get an email, you plug in another drive, click a button in the GUI, and wait for the volume to reshard/resilver/remirror. A freenas box does this as well, but won't have the pretty drive light indicator if it's a home-built box, but then you're not limited to proprietary hardware.

I had been meaning to look into Synology more and I watched a few reviews after this exchange. Sound good except I'm not happy about having to link to their servers. But everything works that way these days :/

Why would you have to? Their devices work fine without any sort of account with them.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#107

Post by watson »

sktrdie wrote: post #101
fenwick67 wrote: post #84

Earlier quoted context omitted.

The big asterisk is, this only works if your database never changes.

What do you mean? The author (say wikipedia owners) can change the db as they usually would change (using UPDATE queries say). Those write queries will result in the least-amount of disk-pages updates. In the torrent world this equals a minimum set of pieces modified and needed to be downloaded by users.

Last I checked you can't update a torrent. So if Wikipedia changes even a single letter, you'd need to download all the data once more

Re: Hyperdrive v10 – a peer-to-peer filesystem

#108

Post by carterschonwald »

smaddox wrote: post #95
sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

It seems like the blog post answers your question pretty thoroughly. The Hyperdrive index and the protocol are tuned for this use case, making it scale to being able to host a Wikipedia clone. BitTorrent FS + SQlite are not tuned for this use case.

Wikipedia’s text history absolutely fits on a tiny hard drive and is easy to get a replica of.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#109

Post by [deleted] »

yjftsjthsd-h wrote: post #74
andrewosh wrote: post #16

Earlier quoted context omitted.

Fair enough -- updated it to POSIX-like. The main point there being that it should be as straightforward to use as Node's fs module.

I don't think that was the humor. Node is, from one point of view, a piece of a browser engine that's been extracted out. And you wrote a filesystem in it. It's like writing a webapp in x86_64 assembly - you can do it, obviously, but it can be viewed as an odd juxtaposition.

[deleted]

Re: Hyperdrive v10 – a peer-to-peer filesystem

#110

Post by namibj »

watson wrote: post #107
sktrdie wrote: post #101

Earlier quoted context omitted.

What do you mean? The author (say wikipedia owners) can change the db as they usually would change (using UPDATE queries say). Those write queries will result in the least-amount of disk-pages updates. In the torrent world this equals a minimum set of pieces modified and needed to be downloaded by users.

Last I checked you can't update a torrent. So if Wikipedia changes even a single letter, you'd need to download all the data once more

Actually, there is a solution against this. Just combine https://www.bittorrent.org/beps/bep_0030.html (Merkle-tree-based hashing) with https://www.bittorrent.org/beps/bep_0039.html (Feed-URL based updates), and in some settings also https://www.bittorrent.org/beps/bep_0047.html (Specifically the padding files, so that flat files inside a torrent can also be efficiently shared in arbitrary combinations of non-partial files.).
Post reply on HN