Live data from Hacker News

Show HN: Noms – A new decentralized database based on ideas from Git

medium.com

161–170 of 180 posts

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#161
post #102

So, i realize this project is early, but it would be EXTREMELY helpful to walk through someone's use case - like, who is the target here? A business analyst who iterates on cleaning / analyzing small excel csvs? Or someone else? After watching the screencast, all I saw was a bunch of commands explained (could have read the docs for that), instead, I'd like to walk through a use-case where this solves someones problem…

CMS

Yup. Their journey started with Camlistore. Which appears to be a nice personal CMS with built-in syncing.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#162
post #156

Earlier quoted context omitted.

I've played with this problem on and off over the last few years. ShareDB[1] is powered by JSON OT[2], in which each change describes the meaning behind what you're trying to do. (For example, 'increment counter' is different from 'change counter from 2 to 3'. They look the same, but behave differently in the case of conflicts). Just knowing what changed often isn't enough to do proper resolution. I've spent years on…

> Just knowing what changed often isn't enough to do proper resolution. This is why my git commit logs are sometimes: perl -p -i -e 's/FOO/BAR/g' $(find src -name "*.[ch]" -print) Having a "high level" description of what changed makes manual merges easier. I wish git had conflict resolution like this. Treating data as an arbitrary sequence of bits is general and correct. Treating data as having a particular format c…

It does? git merge --strategy xyz means git will invoke git-merge-xyz to do the actual conflict resolution. It comes with a variety of built-in ones (see https://git-scm.com/docs/merge-strategies) but you can write more if you have some special-purpose approach you want to use.

Git also has smudge/clean filters if you want to transform your file to a format where line-by-line textual merge is more meaningful

And you can use .gitattributes to make certain extensions/folders/file/whatever default to a certain treatment.

A good example is https://bitbucket.org/sippey/zippey which unpacks zip archives (and hence file formats based on them, like .jar, .docx, etc) to allow the contents within to be tracked in the git repo better. Other custom formats could do something similar...

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#163

Earlier quoted context omitted.

There's a little bit of terminology overloading going on here. In Noms every value has a type. It's an immutable system, so this type just is. The type of `42` is `Number`. The type of `"foobar"` is `String`. The type of `[42,44]` is `List `. And if you add "foo" to that list, the type becomes `List `. We don't try to infer a general database schema from a few instances of data. We just apply this aggregation up the…

Right - the challenge is that with dynamic typing and without schema validation, it's incredibly easy to break any strongly typed client/consuming application. You think you are dealing with a `List `, you have Go/Java/Haskell/whatever apps which are consuming that in a strongly typed fashion using their idiomatic record types, and then suddenly a user accidentally sends in a single value which turns the tree of valu…

i don't understand how we could have gone in the opposite direction.

Schema validation for us is just looking at the type requirements of the dataset and the type of the value and seeing if they are compatible. How can we do that without first knowing the type of the value?

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#164

I had an idea for this with a buddy in college after doing case study research into Git. I've always considered this the next step into a decentralized world outside of code and non-typed "text". I know .csv's where mentioned a few times; are you looking to narrow into a few specific file types for proof of concept?

We have implemented a bunch of importers. One of them is CSV. Take a look at https://github.com/attic-labs/noms/tree/master/samples/go/cs...

We envision there to be tools that work on certain data types (Noms has a full type system), for example an app that displays all geo locations in a dataset.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#166
post #134

Earlier quoted context omitted.

In some glorious future world you might see things like: ``` Download" rel="nofollow">http://www.who.int/tb/country/data/download/en/::case-data/b... the Data ```

Hmm, if you want people to be able to link to Noms datasets on the web, maybe you should switch to using URLs to name the datasets, instead of a two-part identifier with an URL separated from a dataset name by a "::"? Darcs and Git seem to get by more or less with URLs and relative URLs; do you think that cold work for Noms too? The super REST harmonious way to do this would be to define a new media-type for Noms dat…

This is a good point. We never thought of these strings as URLs, but there are places where it would be nice to use them that only want URLs (the href attribute, for example).

The way we have it now is nice in that any valid URL can be used to locate a database. I am loathe to restrict that.

Interesting point though - thank you!

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#167
post #142

I've been wanting something like Noms for a while. Prolly trees sound really promising. In intro.md, you suggest, "If you wanted to find all the people of a particular age AND having a particular hair color, you could construct a second map having type Map >, and intersect the two sets." In that case, how should I keep the two maps in sync? Do I need to atomically update the logic of all the instances of the applicat…

> Do I need to atomically update the logic of all the instances of the application to modify both maps instead of just one? Or do I keep the second map (the hair color index) in a separate index database and update the index whenever I pull changes from a remote database? (What does the API look like for getting notified of new changes that haven't been indexed yet?)

Currently, you have to manually keep an index up to date. But keep in mind that internally this is what all databases are doing -- manually reflecting changes into indexes -- they just hide it from you.

Eventually, we imagine that there will be tools to declare indexes you want to maintain and we'd do it for you. Note that because Noms is good at diffing, calculating the changes that need to be re-indexed comes for free!

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#168

"...inspired by the elegance and power of Git for years.." Definitely powerful, but elegance?

If you ever look into the internals of how Git works, it is beautiful. Yeah, the UI is kinda a mess, but the idea is inspired.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#169
First off... I'm excited to see this project. There's a lot of potential here and this looks like a good implementation of a nice concept. I have at least a bit of authority behind that statement, since a few years ago, I had the opportunity to build something similar (although smaller in ambition.) A couple things to think about:

* Type accretion - This doesn't change the fact that database clients need to be able to accept historical data formats if they need to access historical data. The schema can't be changed for the older data objects without changing the hashes for that data, so there's no way to do something like a schema migration would work in SQL. For simple schema changes like adding fields, this might not be so hard to deal with, but some changes will be structural in nature and change the relative paths between objects. (This adds complexity to the code of database clients, as well as testing effort.)

* Security - Is there a way to secure objects stored within noms? Let's say I store $SECRET into noms and get back a hash. Does it then become the case that every user with access to the database and the hash can now retrieve the $SECRET? What if permissions need to be granted or revoked to a particular object after it's been stored? A field within a particular object? What if an object shouldn't have been stored in the database at all and needs to be obliterated? (This last problem gets worse if the object to be obliterated contains the only path to data that needs to be retained.)

* Performance - The CAS model effectively takes the stored data, runs it through a blender, and returns you a grey goo of hashes...this is good for replication, but it means you can't get much meaningful information out of a hash. This tends to mean a lot of operations like you might find in an old-school navigational database, and a huge dependency on the time to fetch an object given a hash. Indices can help by reducing the complexity of the traversals you need to do, but only if they're current and you have the index you need.

* Data roll off - How do you expire off data so that it doesn't just monotonically increase in volume? Let's say there's an API to mark an object as purgeable, the problem of identifying other purgeable objects turns into effectively a garbage collection process. (git gc, etc.) There's also the issue of the sheer number of objects that can be involved. The system I was involved with had something like 500K objects/day that had to be purged after 120 days in the system. (Total of 60MM objects line and around 6TB or so) Identifying 500K objects to purge and then specifying those to the data layer for action is not necessarily an easy thing....

* Querying - Server side query logic (and an expression language) is basically essential to performance. Otherwise, you wind up with a network round trip for every edge of the graph you follow. Going back to my first point, whatever querying language is used has to be flexible enough to handle a schema that might be varying over time (through schema accretion).

All four of these bullet points are worthy of a great deal more discussion, and I haven't even broached issues around conflict resolution, differencing, UI concerns, etc. I think there are good approaches to managing lots of these issues, but there's a bunch of engineering involved, as well as some close attention to scope and goals...

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#170

What was the thought process behind making Nommy look like he is pretending to give someone a blowjob?

Not here please.

We detached this comment from https://news.ycombinator.com/item?id=12211882 and marked it off-topic.

Post reply on HN