Live data from Hacker News

Hyperdrive v10 – a peer-to-peer filesystem

blog.hypercore-protocol.org

81–90 of 124 posts

Re: Hyperdrive v10 – a peer-to-peer filesystem

#81
post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

That is pretty clever!

Re: Hyperdrive v10 – a peer-to-peer filesystem

#82
post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

Seems like somone should make a frontend for what you just said, especially that last part which would get annoying to do manually.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#84
post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

The big asterisk is, this only works if your database never changes.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#85
So regarding the multi-writer problem, exactly how far away is this from implementing a central (bare) git repository? Seems to me that all you need is some kind of way to take care of authentication and possibly some way to make git's (mostly) append-only data-structure play nice with Hyperdrives'. I realise I'm probably missing some details, so my question is which details?

Re: Hyperdrive v10 – a peer-to-peer filesystem

#86
post #51

Earlier quoted context omitted.

Do you have any good solutions for this exact use case yet?

Checked Tahoe-LAFS, Perkeep, but they are too low level. Haven't tried Resilio Connect yet, although I would prefer something open-sourced. I think we have all the tech needed already (WebRTC, DHT, NAT hole punching, decades of p2p, encryption, onion security, StorJ/Filecoin) etc but what is lacking is dead simple UX and wide support of operating systems - Windows, Mac, Android, iOS, Linux (Raspberry, Synology, cheap…

Syncthing works pretty well here!

Re: Hyperdrive v10 – a peer-to-peer filesystem

#87
post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

That's very interesting, thanks for the links. I'm working on Scuttlebutt (like Dat/Hypercore) and have been working on reimplementing our stack with 'boring' tooling like SQLite and HTTP, and I've been really enjoying it so far.

I'm going to read your blog post now, thanks a lot for the new info.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#88
post #68

Quick note: it would be nice to ship the hyperdrive daemon as something more self-contained than a npm package. I tried to install it and got some c compile errors. Probably my nodejs version is too old, but you want to have the absolute minimum of friction for users to install it. Beaker comes as an appimage for linux, which worked flawlessly at the first try. Maybe do that for the hyperdrive daemon as well?

A Docker image would also allow for us Windows folks to use this, as Docker can map folders between Windows and Linux. Does one exist? If not, I could give it a go :)

Re: Hyperdrive v10 – a peer-to-peer filesystem

#89
post #57

Earlier quoted context omitted.

Okay. Let's consider your comment within the context of the discussion taking place. What's your point?

We're discussing an application written in JavaScript that runs outside a browser. So the point is that it is unsurprising to discuss node. It's like if we discussed writing a network app in Python and someone said that the GIL was a problem. Technically, there are python implementations that don't have that problem, but in practice we would almost certainly be talking about something that would have been using cpyth…

> It's like if we discussed writing a network app in Python and someone said that the GIL was a problem. Technically, there are python implementations that don't have that problem, but in practice we would almost certainly be talking about something that would have been using cpython, so it's relevant.

Not 'technically' at all. If it's normal python code, and the GIL is the problem, then you can change the interpreter and still use the code. But for something built on node, you can't just pull out node.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#90
post #63
post #51

Earlier quoted context omitted.

Checked Tahoe-LAFS, Perkeep, but they are too low level. Haven't tried Resilio Connect yet, although I would prefer something open-sourced. I think we have all the tech needed already (WebRTC, DHT, NAT hole punching, decades of p2p, encryption, onion security, StorJ/Filecoin) etc but what is lacking is dead simple UX and wide support of operating systems - Windows, Mac, Android, iOS, Linux (Raspberry, Synology, cheap…

Yeah there are no turnkey solutions that I'm aware of. Which is why every time there's a thread like this I come to see what the rumpus is. I'm still waiting for a 'Drobo' like device without the proprietary physical layout. A light or dial goes into orange territory, you head to Best Buy or Amazon and buy the biggest drive that doesn't give you sticker shock, you push a button, out pops your worst drive and in goes…

Synology hardware is pretty close to what you're describing. Light on front goes from green to orange and you get an email, you plug in another drive, click a button in the GUI, and wait for the volume to reshard/resilver/remirror.

A freenas box does this as well, but won't have the pretty drive light indicator if it's a home-built box, but then you're not limited to proprietary hardware.

Post reply on HN