Hi all. I'm one of the creators of Noms. Happy to answer any questions!
Hi, is there any place where you talk about how merging of two different branches works?
Show HN: Noms – A new decentralized database based on ideas from Git
91–100 of 180 posts
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#92Earlier quoted context omitted.
Honest answer is: we don't know yet -- we're working our way up from the bottom. But we (cautiously) don't see any reason why the basic design shouldn't scale to very large (e.g. petabyte) datasets, and that is our eventual goal. That said, we do think there are a lot (even maybe the majority) of use cases in the GB-TB range.
Isn't the append-only design unsuitable for scenarios where many updates/deletes are made? If you update/delete 1GB of your 2GB database each day, then after a year the database is 365GB in size, but the live data is only 2GB. I think the git-like features (history, merging) are very helpful for internal work, but when the dataset must be published, I think in most cases only the newest snapshot should be made availa…
It just depends on the details. If you have a dataset in which 50% of values changes every day, and it doesn't compress well, then yeah, your Noms archive of that entire dataset is going to grow quickly.
In such situations, you could either (eventually, when it is implemented) prune old data, or aggregate the changes into bigger blocks.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#93At first glance, this reminds me a little bit of datomic - all data history is preserved/deduplicated, fork/decentralization features. Can you comment on how it compares?
Thanks, we will take that as a compliment.
I feel weird speaking for them, but at a product level, I think it's fair to characterize Datomic as an application database -- competing with things like mongo, mysql, rethink, etc.
While Noms might be a good fit for certain kinds of application databases (cases where history, or sync, is really important) we're really more focused more on archival, version control, and moving data between systems than being an online transactional database.
Also, at a technical level, unless I'm wildly mistaken, I don't believe that Datomic is content-addressed, and I wouldn't call it "decentralized" (though that word is a bit squishy).
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#94I don't want to downplay this idea, it really is nice to see people doing different/unique things with technology. However, 1 question I have is: Couldn't you just put a CSV/JSON file(s) behind VCS? Eg. Drop my CSV/JSON file(s) onto github.com and then it will be version-controlled ?
You can, and people do that today. It has limitations though:
* The data must be sorted in order for Git to provide good diffs
* It does not scale very well. On my machine, Git refuses to diff files over 1GB (maybe there is a setting for that)
* You must clone the entire repository onto your machine to work with it
* There is no programmatic API -- you must work with the data and changes as text and line diffs
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#95GC I can see a shape of solution for since you can use something like a per-object DVVset to determine the minimum set of unresolved histories required to avoid losing data during conflicts while not unnecessarily ballooning the size of the dataset. However, the inner-object conflict-resolution problem seems a lot harder to solve given that there's no obvious join-semilattice for arbitrary fields/data. Can you discus…
> [...] forcing the end-user to manually deal with fixing problems, which is often fraught with pain and peril, and doesn't seem like a strategy that would work for something that's a database (as opposed to something that's a workflow).
Isn't that what (for example) CouchDB does? I believe the reasoning is that conflict resolution is often application specific, so why not deal with it in the application?
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#96Hi all. I'm one of the creators of Noms. Happy to answer any questions!
How would NOMS work for massive amounts of geo-temporal data? with many inserts and queries but few (zero) updates? Efficient queries on geo-temporal keys are useful.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#97Earlier quoted context omitted.
Git is a competitor. It is fairly common to check data (e.g., csv or json files) into Git today. However, this falls down pretty rapidly. In order to get reasonable diffs, the data has to be sorted, and line-oriented. Also Git just doesn't scale well to larger repos or individual objects. Otherwise, we see the competitors as the way that people distribute data today - custom APIs, zip files full of CSV, etc.
How would you say Noms compares to Datomic[1]? Both projects are working on the same idea of representing a database as tree of commits over time. From my quick inspection, it looks like Noms shows some focus towards working in multiple branches, whereas Datomic, at least in its marketing materials, just talks about preserving a single timeline. [1]: http://www.datomic.com/benefits.html
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#98Earlier quoted context omitted.
Yes, we credited bup in various places, such as the design overview: https://github.com/attic-labs/noms/blob/master/doc/intro.md We were also heavily influenced by camlistore (which I hacked on for awhile), irmin, ipfs, and others who have done a lot of interesting work in this space. We do use rolling checksums, but I think we have done some novel work here: https://github.com/attic-labs/noms/blob/master/doc/intro.m…
bup is focused on backups. Would you say that noms is useful as a backup utility? Do you have any mechanisms for securing integrity, specifically repairing the store in case of inconsistencies? Is there any plans to support any data retention policy/functionality?
Noms should be useful as a backup utility, but I'd say it's especially useful for backing up data which is not files. Think about backing up data which you only have access to via API.
You can take the JSON output of an API and drop it into Noms, then do the same thing tomorrow, and Noms will automatically deduplicate the data as well as give you a nice structured API to read and interact with it.
We have an example of this here: https://github.com/attic-labs/noms/tree/master/samples/js/fl... but it's not working atm due to a bug introduced right before launch. You can look at the code though.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#99GC I can see a shape of solution for since you can use something like a per-object DVVset to determine the minimum set of unresolved histories required to avoid losing data during conflicts while not unnecessarily ballooning the size of the dataset. However, the inner-object conflict-resolution problem seems a lot harder to solve given that there's no obvious join-semilattice for arbitrary fields/data. Can you discus…
> [...] forcing the end-user to manually deal with fixing problems, which is often fraught with pain and peril, and doesn't seem like a strategy that would work for something that's a database (as opposed to something that's a workflow). Isn't that what (for example) CouchDB does? I believe the reasoning is that conflict resolution is often application specific, so why not deal with it in the application?
In case of conflicts, CouchDB assumes the most modified branch of the document (i.e., the document with the higher revision number) is the winner. You can resolve the conflict by choosing a different branch/revision manually, but you can also choose to not do anything.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#100GC I can see a shape of solution for since you can use something like a per-object DVVset to determine the minimum set of unresolved histories required to avoid losing data during conflicts while not unnecessarily ballooning the size of the dataset. However, the inner-object conflict-resolution problem seems a lot harder to solve given that there's no obvious join-semilattice for arbitrary fields/data. Can you discus…
> [...] forcing the end-user to manually deal with fixing problems, which is often fraught with pain and peril, and doesn't seem like a strategy that would work for something that's a database (as opposed to something that's a workflow). Isn't that what (for example) CouchDB does? I believe the reasoning is that conflict resolution is often application specific, so why not deal with it in the application?
Riak does this when not using CRDT data types, but the problem is often that there's no clear application or user way to deal with this either most of the time because it requires an awful lot of context in order to make a good decision.
Using the same example of how Git deals with this. Think about times you've gone through a merge conflict process on a chunk of conflicting code where you don't really have any knowledge or context for why the other stuff that's not yours is even there, and say you don't have a way to collaborate with the other developer or someone in leadership to make sense of those parallel efforts. You can only make sane decisions about complex conflict resolution when you have a lot... a lot... of surrounding context and intent. You need extra metadata, and the underlying system needs to expose that to you. That system being your engineering process, manager, co-worker or in this case that system being the database.
Hence my interest in how they're intending to expose this set of concerns to the user.