How do you handle hash collision?
Same way git does?
Show HN: Noms – A new decentralized database based on ideas from Git
41–50 of 180 posts
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#42Can you elaborate a bit on how the hashing and chunking works? There's a rolling hash for determining chunk boundaries, and also SHA-512/256 somewhere. Does the same data chunked differently have a different hash?
Briefly, there are two main hash functions in use in noms.
sha-2 is used to compute the hash of individual chunks. This is the classic use of hashing in content-addressed systems.
We also use a rolling hash to compute chunk boundaries. We do this in the typical way that tools like bup, camlistore, rsync, and others do for large files.
But our observation was that if you squint your eyes, a merkle tree looks a little like a b-tree. So we use a rolling hash to break up huge lists, maps, and sets into trees where nodes are roughly 4KB. So it's a kind of self-balancing, probabilistic, deterministic b-tree thing.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#43Can you elaborate a bit on how the hashing and chunking works? There's a rolling hash for determining chunk boundaries, and also SHA-512/256 somewhere. Does the same data chunked differently have a different hash?
We never chunk the same data differently. An inviolable rule of Noms is that the same logical value is always chunked the same way and always has the same hash.
If I start with integers 1-1000000 and you start with integers 0-999999, and we both make mutations to converge at the same list, we will end up with the exact same tree, with the exact same hashes.
This is what makes efficient synchronization and diff of noms data possible.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#44We've been struggling managing a collection of periodically updated CSVs & binaries over a few GB's in size, we struggled with Git-LFS and gave up, and we were considering (dreading) SVN, this looks really promising. Cheers!
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#45This looks really exciting, congrats to the team for launching! Could you tell us a bit about how this compares to dat? http://dat-data.com/
Dat is (currently) focused on synchronizing files in a peer-to-peer network.
Noms can store files, but it is much more focused on structured data. You put individual values (numbers, strings, rows, structs, etc) into noms, using a type system that noms defines, and this allows you to query, diff, and efficiently update that data.
Also Noms isn't peer-to-peer (although we hypothesize that it could run reasonably on top of an existing network like IPFS).
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#46Can you elaborate a bit on how the hashing and chunking works? There's a rolling hash for determining chunk boundaries, and also SHA-512/256 somewhere. Does the same data chunked differently have a different hash?
We never chunk the same data differently. An inviolable rule of Noms is that the same logical value is always chunked the same way and always has the same hash. If I start with integers 1-1000000 and you start with integers 0-999999, and we both make mutations to converge at the same list, we will end up with the exact same tree, with the exact same hashes. This is what makes efficient synchronization and diff of nom…
Thanks. So it's building a hash tree with deterministic chunking and that means you can cheaply update the hash after updating parts of the tree as you only have to rehash certain bits?
Does that mean that your chunk sizes are kind of fixed? Do you think there's a way to retain that advantage and be able to coalesce smaller chunks into larger ones?
Say your smallest nodes are 4KB but for more efficient storage you might want to go up to 4MB chunks. Could that be done while retaining the same hash for the same underlying data?
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#47Pretty impressive work but seems like reinventing wheels. Why wasn't it built upon existing tech?
I think the docs should enumerate the most important differences and use cases for which it should be a better fit.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#48This is really interesting! What are some ideal use cases for the current implementation? I've seen Git is considered a competitor, but Noms also appears to be a generic database, so i would just like to hear some basic use cases, if possible. Eg: If used as a database, what applications would benefit from Noms? Could/should this be used for personal storage? Could/should this be used for code versioning (ie, Git)?
The way I read it, git is not a competitor but rather an inspiration. They are taking ideas from git to apply them to a different domain.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#49Going through the SDK docs, why was a scheme like 'http://localhost:8000::people' chosen instead of the plain old 'http://localhost:8000/people'? Are there any benefits? If yes, curious to know what they are.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#50Strawman marketing alert: "The most common way to share data today is to post CSV files on a website". Maybe there are a bunch of people that still do that somewhere, but if so, they ain't early adopters of decentralized database technology and so not your target customers. It's always better to talk about what your most likely customers are doing now.
Yea and then a paragraph about git under it, made me double take to think maybe they meant CVS, but then realized they really did mean CSV.
If this is supposed to replace CSV cool, but there is a lot of ways to cross that bridge.
Curious however, what the target use case is. Is it a format, or a database , or both?