Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperdrive v10 – a peer-to-peer filesystem

blog.hypercore-protocol.org

Post reply on HN
Print view View original

91–100 of 124 posts

Re: Hyperdrive v10 – a peer-to-peer filesystem

#91

Post by rrdharan »

fenwick67 wrote: post #84
sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

The big asterisk is, this only works if your database never changes.

Not sure if the parent edited their post after you stated this but note they explained a technique to accommodate database updates / changes / edits.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#93

Post by steve76 »

sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

Wouldn't seeding be a problem? You would need to seed from something that supports webtorrent which uses WebRTC.

With dat-sdk, users just need to go to a webpage. You really just need WebRTC without torrents.

I got rid of multiwriter by just having a dat archive for each user, and the users sharing their dat addresses with each other. They write to their own. When that happens, events emit and users listening write to theirs.

If enough users stay on, listening to each other's address, I only need a web client.

Also, if I have offline support, like Workbox Background Sync, I don't even need internet and information transfers device to device with just an offline PWA. At least that's my goal.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#94

Post by Reelin »

beckingz wrote: post #44
hinkley wrote: post #41

Earlier quoted context omitted.

My go-to example for 'private cloud' (which as soon as we hit the Trough of Disillusionment on this Hype Train, we will all be talking about) is having a set of people in your social or family group maintaining a file share for your semi-private photos and videos. You don't have to worry about a flood or tornado taking out all the photos of first birthdays, or Grandma when she was younger than you are now, and you al…

Do you have any good solutions for this exact use case yet?

Perhaps I'm completely misunderstanding something but don't centralized self hosted cloud-plus-app services address this exact usecase? Things like NextCloud, Seafile, and Sandstorm?

The only major drawback seems to be that you have to host the physical hardware yourself due to lack of solid end-to-end encryption for most platforms. Sandstorm might have it (I'm not clear on what's client- and what's server- side there), Seafile has end-to-end encryption that doesn't protect metadata (https://forum.seafile.com/t/how-strong-is-the-encryption/627...), and NextCloud appears to have a long-running beta of end-to-end encryption on a per-folder basis (https://nextcloud.com/endtoend/). Apparently Cryptomator exists (https://cryptomator.org/) although I've never tried it myself.

If you want to decentralize your shared files unfortunately last I checked SyncThing didn't yet support end-to-end encryption. (I don't think any of the other ones I mentioned can be used in a decentralized manner but things move quickly so I'm not sure.)

Alternatively, if you were thinking more chat and messaging there's self hostable federated services such as PixelFed, PeerTube, Mastodon, and Matrix. Or was there some other usecase you had in mind?

Re: Hyperdrive v10 – a peer-to-peer filesystem

#95

Post by smaddox »

sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

It seems like the blog post answers your question pretty thoroughly. The Hyperdrive index and the protocol are tuned for this use case, making it scale to being able to host a Wikipedia clone. BitTorrent FS + SQlite are not tuned for this use case.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#96

Post by Reelin »

hinkley wrote: post #63
imhoguy wrote: post #51

Earlier quoted context omitted.

Checked Tahoe-LAFS, Perkeep, but they are too low level. Haven't tried Resilio Connect yet, although I would prefer something open-sourced. I think we have all the tech needed already (WebRTC, DHT, NAT hole punching, decades of p2p, encryption, onion security, StorJ/Filecoin) etc but what is lacking is dead simple UX and wide support of operating systems - Windows, Mac, Android, iOS, Linux (Raspberry, Synology, cheap…

Yeah there are no turnkey solutions that I'm aware of. Which is why every time there's a thread like this I come to see what the rumpus is. I'm still waiting for a 'Drobo' like device without the proprietary physical layout. A light or dial goes into orange territory, you head to Best Buy or Amazon and buy the biggest drive that doesn't give you sticker shock, you push a button, out pops your worst drive and in goes…

4 TB drives are now commonly well under $100 (I've seen them as low as $70) and many Micro-ATX and NUC boards have 4+ SATA connectors. Btrfs has CoW, subvolumes, snapshots, and data integrity features.

For the physical device at least, Debian and 2 Btrfs data drives in RAID 1 certainly isn't turnkey but seems quite accessible at this point.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#97

Post by hinkley »

mceachen wrote: post #90
hinkley wrote: post #63

Earlier quoted context omitted.

Yeah there are no turnkey solutions that I'm aware of. Which is why every time there's a thread like this I come to see what the rumpus is. I'm still waiting for a 'Drobo' like device without the proprietary physical layout. A light or dial goes into orange territory, you head to Best Buy or Amazon and buy the biggest drive that doesn't give you sticker shock, you push a button, out pops your worst drive and in goes…

Synology hardware is pretty close to what you're describing. Light on front goes from green to orange and you get an email, you plug in another drive, click a button in the GUI, and wait for the volume to reshard/resilver/remirror. A freenas box does this as well, but won't have the pretty drive light indicator if it's a home-built box, but then you're not limited to proprietary hardware.

I had been meaning to look into Synology more and I watched a few reviews after this exchange. Sound good except I'm not happy about having to link to their servers. But everything works that way these days :/

Re: Hyperdrive v10 – a peer-to-peer filesystem

#98

Post by gnu »

jhardy54 wrote: post #87
sktrdie wrote: post #76

This is really cool, but why reinvent the wheel? For instance SQLite already has tons of years of optimization regarding storing and accessing files on disk. To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it. Users can use sqltorrent Virtual File System ( https://github.com/bittorrent/sqltorren…

That's very interesting, thanks for the links. I'm working on Scuttlebutt (like Dat/Hypercore) and have been working on reimplementing our stack with 'boring' tooling like SQLite and HTTP, and I've been really enjoying it so far. I'm going to read your blog post now, thanks a lot for the new info.

Interesting! Is there any document that I can read about your reimplementation? Or any code?

Re: Hyperdrive v10 – a peer-to-peer filesystem

#99

Post by andrewosh »

andrewosh wrote: post #32
Mizza wrote: post #27

Been following this for years, congratulations on your release. I looked at dat for building P2P services, but found that ZeroNet was actually more capable for "real" services - maybe now is the time to reevaluate that, especially given the improvements to large (both deep and wide) archives and the new hole punching. Can you please mention your thoughts on: - Discoverability of content in Hyperswarm (DHT Search/"Sup…

Thanks Mizza! Not sure if I'm answering your first point, but Hyperswarm is baked into the Hyperdrive daemon, so daemon users should have their drives swarmed/available automatically. There are a few CLI commands to toggle this behavior too, in case you don't want to add your drive's key to the DHT. The Hypercore Protocol org's creating a similar proposal repo called HYP [0] (we couldn't resist the name), scoped tigh…

In the flurry of the launch I got all my links wrong. The HYP repo can be found here: https://github.com/hypercore-protocol/hyp

Re: Hyperdrive v10 – a peer-to-peer filesystem

#100

Post by hutzlibu »

This sounds good!

But I would not call it finished, as of today, apparently only one person can make changes to the filesystem. That does limit the use cases.

"In v10, we don't go all the way to a general multi-writer solution; solving multi-writer scalably, without incurring major performance penalties or confusing UX, remains a research question for us. "

.. so, give them some support, so they can solve this.

Post reply on HN