Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Noms – A new decentralized database based on ideas from Git

medium.com

Post reply on HN
Print view View original

131–140 of 180 posts

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#131

Post by fiatjaf »

im_down_w_otp wrote: post #104
fiatjaf wrote: post #99

Earlier quoted context omitted.

In case of conflicts, CouchDB assumes the most modified branch of the document (i.e., the document with the higher revision number) is the winner. You can resolve the conflict by choosing a different branch/revision manually, but you can also choose to not do anything.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

It is just that random if the systems remain isolated for a long time. Since CouchDB requires you to send the last revision number when updating a document, if the systems are are live replicating between themselves, the guy who is hitting the queue more rapidly will be forced to fetch the latest winning revision every time before hitting the queue (that may be a revision from a different guy). This will give him time to think about the revision he just received, perhaps examine the document linked to that revision number, see if everything is place, perhaps merge changes himself manually... all that before updating the document in the database.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#132

Post by im_down_w_otp »

fiatjaf wrote: post #131
im_down_w_otp wrote: post #104

Earlier quoted context omitted.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

It is just that random if the systems remain isolated for a long time. Since CouchDB requires you to send the last revision number when updating a document, if the systems are are live replicating between themselves, the guy who is hitting the queue more rapidly will be forced to fetch the latest winning revision every time before hitting the queue (that may be a revision from a different guy). This will give him tim…

This seems to only make sense in the case that I'm sure I've been able to read the most recent writes from all replicas, no?

I feel like there should be some other method to carry on this discussion besides this thread about Noms :-)

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#133

Post by Lazare »

im_down_w_otp wrote: post #104
fiatjaf wrote: post #99

Earlier quoted context omitted.

In case of conflicts, CouchDB assumes the most modified branch of the document (i.e., the document with the higher revision number) is the winner. You can resolve the conflict by choosing a different branch/revision manually, but you can also choose to not do anything.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

> How is, "whoever hits the queue most often" a useful deterministic resolution strategy?

It's a deterministic resolution strategy, and is thus useful.

> I guess it's functionally no worse than wall-clock time or something,

Wall-clock time is not deterministic; therefore it's far worse.

When dealing with distributed systems, deterministic processes are critical. Multiple systems all being right is awesome, but multiple systems being wrong in different ways is a nightmare. :)

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#134

Post by kragen »

aboodman wrote: post #75
spotman wrote: post #66

Earlier quoted context omitted.

So if nom takes off would you see 'download a nom dataset by clicking here' or would it be 'use this hostname to sync the nom to your own computer' Or would it be a dsn sort of and you just instantiate a client and your on your way? Or, some combo?

In some glorious future world you might see things like: ``` Download" rel="nofollow">http://www.who.int/tb/country/data/download/en/::case-data/b... the Data ```

Hmm, if you want people to be able to link to Noms datasets on the web, maybe you should switch to using URLs to name the datasets, instead of a two-part identifier with an URL separated from a dataset name by a "::"? Darcs and Git seem to get by more or less with URLs and relative URLs; do you think that cold work for Noms too?

The super REST harmonious way to do this would be to define a new media-type for Noms databases with a smallish document that links to the component parts. Like torrent files, but using URLs (maybe relative URLs) instead of SHA1 hashes for the components, maybe?

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#135

Post by im_down_w_otp »

Lazare wrote: post #133
im_down_w_otp wrote: post #104

Earlier quoted context omitted.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

> How is, "whoever hits the queue most often" a useful deterministic resolution strategy? It's a deterministic resolution strategy, and is thus useful. > I guess it's functionally no worse than wall-clock time or something, Wall-clock time is not deterministic; therefore it's far worse. When dealing with distributed systems, deterministic processes are critical. Multiple systems all being right is awesome, but multip…

Is it deterministic in a way that's useful? From the perspective of the end-user its going to appear random because they don't control the system environment where "highest revision number" can mean something useful to them. In fact, the CouchDB guide even alludes to this when they talk about not relying on this scheme for complex conflict resolution it seems.

Two nodes split. A and B. Say there are 100 updates to A and 500 updates to B. The split heals, the system picks B because 500 > 100, but the write you actually want to dominate is A. The user can't control which replica gets hit more often, or when a split happens, so while this might be deterministic inside the DB it is semantically random from the user's perspective. So the system can make the same choice on all replicas, assuming it can guarantee it has seen all replicas, which I guess allows you to push merging management to each replica instead of requiring an intermediate coordination replica and then re-publishing the merge state to the replicas? So there's a system optimization benefit there.

But consider if the system did pick a winner at random, how would this look any different to the user? The user doesn't necessarily know if A or B should be picked.

Deterministic behavior is really important, but it seems like it really only looks non-random to the end user when deterministically picking a least upper bound for converging a join-semilattice or when all operations on the data are commutative or idempotent doesn't it?

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#137

Post by kragen »

aboodman wrote: post #126
alexatkeplar wrote: post #76

Earlier quoted context omitted.

Why would you exclusively support schema inference, rather than also allowing users to manually specify their schemas? Schema inference is very difficult to do correctly and safely, especially with small initial samples of instances (source: work on https://github.com/snowplow/schema-guru ).

There's a little bit of terminology overloading going on here. In Noms every value has a type. It's an immutable system, so this type just is. The type of `42` is `Number`. The type of `"foobar"` is `String`. The type of `[42,44]` is `List `. And if you add "foo" to that list, the type becomes `List `. We don't try to infer a general database schema from a few instances of data. We just apply this aggregation up the…

It sounds like Noms is dynamically typed, rather like SQLite — types are associated with values, not (just) with datasets. The difference is that SQLite (like Python or JS) only types leaf/atomic data, while you're also typing aggregate data. Is that right?

Are you planning on writing complete reference documentation at some point, like https://www.sqlite.org/limits.html, https://www.sqlite.org/howtocorrupt.html, https://www.sqlite.org/lang.html, https://docs.python.org/2/reference/index.html, and https://golang.org/ref/spec? Or is using Noms going to be more of a UTSL kind of thing? The documentation I've found so far seems to be purely tutorial and introductory in nature.

(I'm really glad you're writing Noms, by the way. There's an enormous need for it.)

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#138

Post by robzyb »

Wow, this could be quite interesting.

Firstly, it would be cool if this could be a single gateway to "all the data in the world". Right now its a pain to find, say, energy generation statistics for, say, Portugal, but it would be great if I could do something like:

  noms get statistics.industry.energy.portugal.all();
Secondly, the versioning idea could have some really cool applications. For example, I work in data analytics, and sometimes I want to transform some data in an SQL table.

Doing transformations nicely is a bit difficult. Either I'm doing the calculations in a column of a view, with the associated performance hit, or I'm tacking columns onto the table, which quickly leads to a mess, especially during the initial stages of analyses.

It would be so cool if I could treat the database as a constantly-evolving git tree.

Post reply on HN