So, i realize this project is early, but it would be EXTREMELY helpful to walk through someone's use case - like, who is the target here? A business analyst who iterates on cleaning / analyzing small excel csvs? Or someone else? After watching the screencast, all I saw was a bunch of commands explained (could have read the docs for that), instead, I'd like to walk through a use-case where this solves someones problem…
CMS
Show HN: Noms – A new decentralized database based on ideas from Git
161–170 of 180 posts
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#162Earlier quoted context omitted.
I've played with this problem on and off over the last few years. ShareDB[1] is powered by JSON OT[2], in which each change describes the meaning behind what you're trying to do. (For example, 'increment counter' is different from 'change counter from 2 to 3'. They look the same, but behave differently in the case of conflicts). Just knowing what changed often isn't enough to do proper resolution. I've spent years on…
> Just knowing what changed often isn't enough to do proper resolution. This is why my git commit logs are sometimes: perl -p -i -e 's/FOO/BAR/g' $(find src -name "*.[ch]" -print) Having a "high level" description of what changed makes manual merges easier. I wish git had conflict resolution like this. Treating data as an arbitrary sequence of bits is general and correct. Treating data as having a particular format c…
Git also has smudge/clean filters if you want to transform your file to a format where line-by-line textual merge is more meaningful
And you can use .gitattributes to make certain extensions/folders/file/whatever default to a certain treatment.
A good example is https://bitbucket.org/sippey/zippey which unpacks zip archives (and hence file formats based on them, like .jar, .docx, etc) to allow the contents within to be tracked in the git repo better. Other custom formats could do something similar...
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#163Earlier quoted context omitted.
There's a little bit of terminology overloading going on here. In Noms every value has a type. It's an immutable system, so this type just is. The type of `42` is `Number`. The type of `"foobar"` is `String`. The type of `[42,44]` is `List `. And if you add "foo" to that list, the type becomes `List `. We don't try to infer a general database schema from a few instances of data. We just apply this aggregation up the…
Right - the challenge is that with dynamic typing and without schema validation, it's incredibly easy to break any strongly typed client/consuming application. You think you are dealing with a `List `, you have Go/Java/Haskell/whatever apps which are consuming that in a strongly typed fashion using their idiomatic record types, and then suddenly a user accidentally sends in a single value which turns the tree of valu…
Schema validation for us is just looking at the type requirements of the dataset and the type of the value and seeing if they are compatible. How can we do that without first knowing the type of the value?
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#164I had an idea for this with a buddy in college after doing case study research into Git. I've always considered this the next step into a decentralized world outside of code and non-typed "text". I know .csv's where mentioned a few times; are you looking to narrow into a few specific file types for proof of concept?
We envision there to be tools that work on certain data types (Noms has a full type system), for example an app that displays all geo locations in a dataset.
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#165Re: Show HN: Noms – A new decentralized database based on ideas from Git
#166Earlier quoted context omitted.
In some glorious future world you might see things like: ``` Download" rel="nofollow">http://www.who.int/tb/country/data/download/en/::case-data/b... the Data ```
Hmm, if you want people to be able to link to Noms datasets on the web, maybe you should switch to using URLs to name the datasets, instead of a two-part identifier with an URL separated from a dataset name by a "::"? Darcs and Git seem to get by more or less with URLs and relative URLs; do you think that cold work for Noms too? The super REST harmonious way to do this would be to define a new media-type for Noms dat…
The way we have it now is nice in that any valid URL can be used to locate a database. I am loathe to restrict that.
Interesting point though - thank you!
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#167I've been wanting something like Noms for a while. Prolly trees sound really promising. In intro.md, you suggest, "If you wanted to find all the people of a particular age AND having a particular hair color, you could construct a second map having type Map >, and intersect the two sets." In that case, how should I keep the two maps in sync? Do I need to atomically update the logic of all the instances of the applicat…
Currently, you have to manually keep an index up to date. But keep in mind that internally this is what all databases are doing -- manually reflecting changes into indexes -- they just hide it from you.
Eventually, we imagine that there will be tools to declare indexes you want to maintain and we'd do it for you. Note that because Noms is good at diffing, calculating the changes that need to be re-indexed comes for free!
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#168"...inspired by the elegance and power of Git for years.." Definitely powerful, but elegance?
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#169* Type accretion - This doesn't change the fact that database clients need to be able to accept historical data formats if they need to access historical data. The schema can't be changed for the older data objects without changing the hashes for that data, so there's no way to do something like a schema migration would work in SQL. For simple schema changes like adding fields, this might not be so hard to deal with, but some changes will be structural in nature and change the relative paths between objects. (This adds complexity to the code of database clients, as well as testing effort.)
* Security - Is there a way to secure objects stored within noms? Let's say I store $SECRET into noms and get back a hash. Does it then become the case that every user with access to the database and the hash can now retrieve the $SECRET? What if permissions need to be granted or revoked to a particular object after it's been stored? A field within a particular object? What if an object shouldn't have been stored in the database at all and needs to be obliterated? (This last problem gets worse if the object to be obliterated contains the only path to data that needs to be retained.)
* Performance - The CAS model effectively takes the stored data, runs it through a blender, and returns you a grey goo of hashes...this is good for replication, but it means you can't get much meaningful information out of a hash. This tends to mean a lot of operations like you might find in an old-school navigational database, and a huge dependency on the time to fetch an object given a hash. Indices can help by reducing the complexity of the traversals you need to do, but only if they're current and you have the index you need.
* Data roll off - How do you expire off data so that it doesn't just monotonically increase in volume? Let's say there's an API to mark an object as purgeable, the problem of identifying other purgeable objects turns into effectively a garbage collection process. (git gc, etc.) There's also the issue of the sheer number of objects that can be involved. The system I was involved with had something like 500K objects/day that had to be purged after 120 days in the system. (Total of 60MM objects line and around 6TB or so) Identifying 500K objects to purge and then specifying those to the data layer for action is not necessarily an easy thing....
* Querying - Server side query logic (and an expression language) is basically essential to performance. Otherwise, you wind up with a network round trip for every edge of the graph you follow. Going back to my first point, whatever querying language is used has to be flexible enough to handle a schema that might be varying over time (through schema accretion).
All four of these bullet points are worthy of a great deal more discussion, and I haven't even broached issues around conflict resolution, differencing, UI concerns, etc. I think there are good approaches to managing lots of these issues, but there's a bunch of engineering involved, as well as some close attention to scope and goals...
Re: Show HN: Noms – A new decentralized database based on ideas from Git
#170What was the thought process behind making Nommy look like he is pretending to give someone a blowjob?
We detached this comment from https://news.ycombinator.com/item?id=12211882 and marked it off-topic.