71–80 of 124 posts
1. Consensus. If I submit conflicting updates and sign both, how does the swarm resolve what is the latest state?
2. Migrating the swarm. If all the machines in a swarm get corrupted, can I migrate to a totally new swarm?
How do you handle merge conflicts? Since you aren't calling it a blockchain I take it that Hypercore is targeted at the trusted (writers) and semi-trusted (readers) use case?
Hypercore is a single writer append-only log. The website has a bit more info about how it works, but's basically a merkle log signed by a private key / public key scheme. We build collaborative data structures by combining multiple Hypercores. Hyperdrives builds a p2p filesystem on top of Hypercore for a single writer. Using mounts you can mount other peoples drives so merge conflicts don't happen since there is no…
Do you agree that for the collaborative data structures side of things (like the chat app) users of the hypercore-protocol will likely run into clock trust problems?
PS I'm a big fan of your work/repos.
Earlier quoted context omitted.
Hypercore is a single writer append-only log. The website has a bit more info about how it works, but's basically a merkle log signed by a private key / public key scheme. We build collaborative data structures by combining multiple Hypercores. Hyperdrives builds a p2p filesystem on top of Hypercore for a single writer. Using mounts you can mount other peoples drives so merge conflicts don't happen since there is no…
So it sounds a bit like you've replicated git. Do you agree that for the collaborative data structures side of things (like the chat app) users of the hypercore-protocol will likely run into clock trust problems? PS I'm a big fan of your work/repos.
Oh dude, I'm sorry but I read > Hyperdrive is a POSIX-compatible filesystem implementation, written in Node.js And just cracked up! Thanks, I needed that.
Fair enough -- updated it to POSIX-like. The main point there being that it should be as straightforward to use as Node's fs module.
Earlier quoted context omitted.
These days, using JavaScript outside of the browser sort of implies Node.js, in the same way that using Python implies that you're using CPython, unless you specify otherwise. In theory your distinction makes sense, but in practice, very few people are using anything but Node.js for backend JavaScript systems (at least for new projects).
Okay. Let's consider your comment within the context of the discussion taking place. What's your point?
To make SQLite decentralized (like Hyperdrive) you can put in a torrent. Index it using full-text-search https://sqlite.org/fts5.html for instance. Then let the users seed it.
Users can use sqltorrent Virtual File System (https://github.com/bittorrent/sqltorrent) to query the db without downloading the entire torrent - essentially it knows to download only the pieces of the torrent to satisfy the query. This is similar techniques behind Hyperdrive I believe just again, using standard tools and tech that exists and highly optimized: https://www.sqlite.org/vfs.html
Every time a new version of the SQLite db is published (say by wikipedia), the peers can change to the new torrent and reuse the pieces they already have - since SQLite is indexed in an optimal way to reduce file changes (and hence piece changes) when the data is updated.
I talk a bit about it here: https://medium.com/@lmatteis/torrentnet-bd4f6dab15e4
Again not against redoing things better, but why not use existing proven tech for certain parts of the tool?
Now, onto the content, you touch on de-duplication. I am quite concerned with the cost associated to updating a large file. Is something like rolling hashes investigated, to chunk files independently of their size? I guess it is, given you seem to be working hard on de-duplication.
But then, that kind of trick best works on uncompressed data, which is inefficient for transmission. Is data compressed before transmission? Whole chunks, or whole files at a time? Ahead of time? Interactively based on what the peer needs?
The trade-offs are many, and complex to investigate. I'm wondering if this could be used as an OS image, like OSTree does?
And lastly, I did not get if multiple peers having the same private key identity could modify the structure simultaneously. What would happen?
Also, nodejs gave me a kneejerk reaction that may be unwarranted, but that's quite a huge dependency to pull in for something that wants to be a ubiquitous building block. Does it have a C API? Also, twitter, discord, github (node to a lesser extent)... it seems somewhat ironic to build a ultimate decentralized filesystem while relying on these hypercentralized offerings, and I am afraid it could turn some contributors off.
Interesting, thanks. Not clear to me when reading the announcement is: will hyperdrive replace dat? From reading the comments here, it seems that the answer is "yes". Now, onto the content, you touch on de-duplication. I am quite concerned with the cost associated to updating a large file. Is something like rolling hashes investigated, to chunk files independently of their size? I guess it is, given you seem to be wo…
But dat/hyperdrive is a well-documented protocol, and there are several ports to other languages.
I am very interested in the rust port, but I am not sure in what state it is: https://datrs.yoshuawuyts.com/
I have seen some tweets about progress being made on this. Does anybody know more?
Interesting, thanks. Not clear to me when reading the announcement is: will hyperdrive replace dat? From reading the comments here, it seems that the answer is "yes". Now, onto the content, you touch on de-duplication. I am quite concerned with the cost associated to updating a large file. Is something like rolling hashes investigated, to chunk files independently of their size? I guess it is, given you seem to be wo…
I kinda share your kneejerk reaction about nodejs for foundational tools, even though I must admit that dat/hyperdrive works surprisingly well for a js tool. But dat/hyperdrive is a well-documented protocol, and there are several ports to other languages. I am very interested in the rust port, but I am not sure in what state it is: https://datrs.yoshuawuyts.com/ I have seen some tweets about progress being made on th…
The wire protocol works now: https://github.com/Frando/hypercore-protocol-rs and the community is active in #datrs on freenode
Earlier quoted context omitted.
Okay. Let's consider your comment within the context of the discussion taking place. What's your point?
We're discussing an application written in JavaScript that runs outside a browser. So the point is that it is unsurprising to discuss node. It's like if we discussed writing a network app in Python and someone said that the GIL was a problem. Technically, there are python implementations that don't have that problem, but in practice we would almost certainly be talking about something that would have been using cpyth…
First, this is very much being developed in a similar scope the millions of lines of JS that are in Firefox. This is going into Beaker. See beakerbrowser.com. Just as a point of fact.
Secondly, you still haven't connected your observations to the discussion! That observation amounts to, hey, there's lots of JS written for the NodeJS ecosystem—most of it is, really. And once again, my response is, "Okay, so it exists. So what?" I'm truly struggling to understand the significance of the comments here. Like, what if anything is that supposed to change? What is anyone supposed to do with that information? Is it supposed to change someone's mind? Is it supposed to change mine? And if so, in reference to what specifically? Is it even new information?
The most meaningful thing that I can manage to parse is where you say "the point is that it is unsurprising to discuss node". But whoever said it was surprising? Do you think I'm surprised? choeger mentioned NodeJS by name. My comment is evidence of a nuanced understanding of NodeJS. Where's the surprise?