Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperdrive v10 – a peer-to-peer filesystem

blog.hypercore-protocol.org

Post reply on HN
Print view View original

51–60 of 124 posts

Re: Hyperdrive v10 – a peer-to-peer filesystem

#51

Post by imhoguy »

beckingz wrote: post #44
hinkley wrote: post #41

Earlier quoted context omitted.

My go-to example for 'private cloud' (which as soon as we hit the Trough of Disillusionment on this Hype Train, we will all be talking about) is having a set of people in your social or family group maintaining a file share for your semi-private photos and videos. You don't have to worry about a flood or tornado taking out all the photos of first birthdays, or Grandma when she was younger than you are now, and you al…

Do you have any good solutions for this exact use case yet?

Checked Tahoe-LAFS, Perkeep, but they are too low level. Haven't tried Resilio Connect yet, although I would prefer something open-sourced.

I think we have all the tech needed already (WebRTC, DHT, NAT hole punching, decades of p2p, encryption, onion security, StorJ/Filecoin) etc but what is lacking is dead simple UX and wide support of operating systems - Windows, Mac, Android, iOS, Linux (Raspberry, Synology, cheap VPS backup)

Re: Hyperdrive v10 – a peer-to-peer filesystem

#54

Post by andrewosh »

obiefernandez wrote: post #52

How soon until this can replace Dropbox for my little distributed working group consisting of 3 people wanting to share a single file system?

Our little team has been doing just this, both for dogfooding and because it works well for us.

Take a look at the "mounts" section of the blog post where we describe a group pattern you can set up. You can create a group directory called "team-drive", then mount each user's drive into the group.

If you're using FUSE, this will feel similar to Dropbox, but with one directory per-user.

We're starting with these kinds of simple mounts, and brainstorming ways to extend them soon.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#55

Post by newman314 »

imhoguy wrote: post #51
beckingz wrote: post #44

Earlier quoted context omitted.

Do you have any good solutions for this exact use case yet?

Checked Tahoe-LAFS, Perkeep, but they are too low level. Haven't tried Resilio Connect yet, although I would prefer something open-sourced. I think we have all the tech needed already (WebRTC, DHT, NAT hole punching, decades of p2p, encryption, onion security, StorJ/Filecoin) etc but what is lacking is dead simple UX and wide support of operating systems - Windows, Mac, Android, iOS, Linux (Raspberry, Synology, cheap…

I use Resilio Sync to sync my 1Password. It mostly works but is clunky in that the handling of identities is clumsy (unable to cleanly remove) as well as having less than stellar UX for adding/removing folders.

Still looking for the perfect Dropbox-like experience but without the cloud sync piece. If only syncthing had decent mobile apps...

Re: Hyperdrive v10 – a peer-to-peer filesystem

#56

Post by rklaehn »

mafintosh wrote: post #50
rklaehn wrote: post #49

Congrats for the release. Mostly familiar with IPFS, but learned about DAT at dtn.is last year. The hypercore protocol is pretty well designed and documented. How would hyperdrive deal with the following scenario: you got a large dataset such as wikipedia. Lots of people have browsed it, but most of them only have a tiny fraction. How do yo know which peers to connect to to get a particular bit (offset?) you are inte…

The peers gossip using compressed bitfields in regards to what data they have. These bitfields are super small so we can pack quite a bit of information. At the moment we don't do anything special in regards to discovery, but as we scale that's something we want to investigate. Since everything is running on append-only logs we can group the data into sections quite easily so there is some easy wins we can do there w…

So do you opportunistically gather info about what other peers have via this gossip protocol, or just when you need something?

I had looked at https://datprotocol.github.io/how-dat-works/ , but I don't remember anything about a gossip protocol or a peer building a "view of the world". Is that new?

Re: Hyperdrive v10 – a peer-to-peer filesystem

#57

Post by cxr »

RussianCow wrote: post #45
cxr wrote: post #30

Earlier quoted context omitted.

JavaScript is a programming language. NodeJS is not. It's the substrate that the NPM ecosystem runs on. choeger's comment very well may have not been made with a distinction in mind (and I agree that it was a low effort, low signal comment), but the distinction exists. choeger seems to be using Firefox. Firefox is implemented using an Emacs-like architecture. The Firefox codebase includes millions of lines of JS, and…

These days, using JavaScript outside of the browser sort of implies Node.js, in the same way that using Python implies that you're using CPython, unless you specify otherwise. In theory your distinction makes sense, but in practice, very few people are using anything but Node.js for backend JavaScript systems (at least for new projects).

Okay. Let's consider your comment within the context of the discussion taking place. What's your point?

Re: Hyperdrive v10 – a peer-to-peer filesystem

#59

Post by mceachen »

beckingz wrote: post #44
hinkley wrote: post #41

Earlier quoted context omitted.

My go-to example for 'private cloud' (which as soon as we hit the Trough of Disillusionment on this Hype Train, we will all be talking about) is having a set of people in your social or family group maintaining a file share for your semi-private photos and videos. You don't have to worry about a flood or tornado taking out all the photos of first birthdays, or Grandma when she was younger than you are now, and you al…

Do you have any good solutions for this exact use case yet?

I'm hoping PhotoStructure solves this usecase when I add sharing in the next version. Note that PhotoStructure is focused on just photos and videos--it isn't a general file sharing solution.

I'm looking into integrating with DAT or Hyperdrive-like solutions to help make storage backups less fiddly for users.

For now, I recommend my beta users use SyncThing or Resilio Sync to get their photos and videos off their phones and on to their home NAS or computer.

Re: Hyperdrive v10 – a peer-to-peer filesystem

#60

Post by mafintosh »

rklaehn wrote: post #56
mafintosh wrote: post #50

Earlier quoted context omitted.

The peers gossip using compressed bitfields in regards to what data they have. These bitfields are super small so we can pack quite a bit of information. At the moment we don't do anything special in regards to discovery, but as we scale that's something we want to investigate. Since everything is running on append-only logs we can group the data into sections quite easily so there is some easy wins we can do there w…

So do you opportunistically gather info about what other peers have via this gossip protocol, or just when you need something? I had looked at https://datprotocol.github.io/how-dat-works/ , but I don't remember anything about a gossip protocol or a peer building a "view of the world". Is that new?

It's only between the peers in your subset of the swarm for now. They exchange a series of WANT and HAVE messages where they subscribe to the sections of each others logs they are interested in.

We are working on expanding this scheme so peers can help discover peers that have the section you are looking for.

Due to the compressed bitfields these section are quite large. In most cases using a few kilobytes you can share WANT/HAVE for millions of blocks

Post reply on HN