Noms – A versioned, forkable, syncable database
1–10 of 107 posts
Re: Noms – A versioned, forkable, syncable database
#2Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
Re: Noms – A versioned, forkable, syncable database
#3Open-source tech like this is nice. This could be used to build a distributed document editing application, for example. Or any application where you want to spin off multiple instances and reconcile the data later.
EDIT: At least one team is investigating layering Noms on top of IPFS [1]. I guess the idea would be to construct something similar to GitTorrent [2]; layering various version-controlled datastores on various p2p protocols could result in several viable architectures.
[1] https://github.com/attic-labs/noms/issues/2123#issuecomment-...
Re: Noms – A versioned, forkable, syncable database
#4Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
Re: Noms – A versioned, forkable, syncable database
#5A couchbase killer.
Re: Noms – A versioned, forkable, syncable database
#6Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
So.. CRDTs?
Re: Noms – A versioned, forkable, syncable database
#7Adam Leventhal (DTrace, OpenZFS) took a look at building a FUSE filesystem on Noms using Go.
Re: Noms – A versioned, forkable, syncable database
#8Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
Which ones are you thinking of?
The nice thing about git is that it doesn't really require much of the server. I run my own "git hosting" with linode, apt-get, and ssh.
Most of the solutions for immutable big data don't have that level of convenience, as far as I know. Anything which requires a lot of sysadmin and ops work will eventually become a commercial cloud service... git on the other hand is useful by itself, and good companies can obviously be built on top of it as well.
Re: Noms – A versioned, forkable, syncable database
#9Very interesting, I think we need a git for data. What is the performance of diffs and merges? What data size does it become too slow?
If by "Git for data" you mean accumulate-only (or append-only), immutable data stores... There are already many existing solutions. It's always good to see alternatives, though!
No not really. CVS and git are basically the same thing but git is a lot better in many ways. I don't know if noms is really the git for data but those other tools are more like CVS in this analogy: clunky and slow.
Re: Noms – A versioned, forkable, syncable database
#10Noms is a great example of the power of decentralized database technology, the interesting research that goes into such systems, and wonderful documentation to browse.
I do want to note some tradeoffs with Content-Addressed and Append-Only systems, as my work on a similar project ( an Open Source Firebase, https://github.com/amark/gun ) made me move away from those ideas (even though they are great ideas).
- Content-Addressed stores are going to revolutionize data integrity and efficiency. But they do have a trade off, it makes it a lot harder to read the data if you do not already know the data you are trying to read! The bottom of the repo metions for instance that a query system has not yet been built. From my experience the reason why is because it is difficult to build query systems on Content-Addressed stores, which is a tradeoff from all the gains you can get from it.
- Append-Only gives you rich features like offline-first support and (if implemented) lovely things like rewind/fastforward data time travel. All very cool. However, do not forget that this then also makes it difficult for you to retrieve the latest whole snapshot of your data. So you are not going to get the read performance that you could.
But the only possible way for us as a community, and people playing around with databases, can figure out what the best system is - is for people to build and experiment. Which is part of the reason why Nom is so cool. It is an invitation to others to actually join, play, and experiment with database technology in an open and encouraging environment. That is incredibly valuable and needed!