Show HN: Noms – A new decentralized database based on ideas from Git • Hacker News Board

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#111

Post by iso-8859-1 » Tue, Aug 02, 2016, 11:47 PM UTC

hosh wrote: post #84

bkalman wrote: post #74

Earlier quoted context omitted.

Say we did just do / . What does the path " http://demo.noms.io/cli-tour/sf-fire-inspections/raw" refer to? Is the database " http://demo.noms.io" and the dataset "cli-tour/sf-fire-inspections/raw"? Is the database " http://demo.noms.io/cli-tour/sf-fire-inspections" and the dataset "raw"? In our sample data (see https://github.com/attic-labs/noms/blob/master/doc/cli-tour.... for example) we actually have this exact p…

There are issues with using `:` in an URL, if you plan on using the URL in a way that's compatible with the extant software out there. I remember: - I remember the Rails community trying to use `;` which broke Mongrel 1. Mongrel's parser was generated from the RFC. There was a huge flame war about that back in the day. The Rails core team at the time thought that Mongrel should make an exception to a reserved charact…

Java breaks:

groovy -e "new URL('http://localhost:8000::people')" Caught: java.net.MalformedURLException

Python breaks:

>>> urlparse('http://localhost:8000::people') ParseResult(scheme='http', netloc='localhost:8000::people', path='', params='', query='', fragment='')

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#112

Post by espadrine » Wed, Aug 03, 2016, 12:00 AM UTC

rafaelweinstein wrote: post #106

im_down_w_otp wrote: post #77

GC I can see a shape of solution for since you can use something like a per-object DVVset to determine the minimum set of unresolved histories required to avoid losing data during conflicts while not unnecessarily ballooning the size of the dataset. However, the inner-object conflict-resolution problem seems a lot harder to solve given that there's no obvious join-semilattice for arbitrary fields/data. Can you discus…

This is a question that we've gotten quite a bit. It's our view that there's no magic solution to conflicts. There are logical conflicts in the real world that must be arbitrated. That said, it's a surprisingly basic thing, but just knowing what changed from party (a) and party (b)'s perspective (relative to their most recently agreed-upon state) is somewhat rare or ad-hoc in existing systems. In noms, you can direct…

To me, there are two separate concerns: intent preservation and coherence.

Operations cause a change of state. The sequence of operations that are performed given a user action need to ensure that they will cause a modification of the state of the database in the way intended by the user.

Coherence is about maintaining a design. A set of rules are conceived. For each state, it is possible to determine whether the state is valid according to them, and if so, the database is coherent.

To maintain coherence, it is possible to deny an operation, therefore breaking user intent. To maintain user intent, it is possible to accept an operation that leads to an invalid state. From what I understand, noms is heavily tilted towards coherence, like git, but unlike git, it doesn't have the social "pull request" aspect, nor a support for running tests.

You mention diffing as a plus, but realistically the only use of diffing is to guess intent. Real intent can only be provided by a maximally rich set of operations. Logs of SQL queries, for instance, are more likely to provide insight than a cold diff. For a database that is tilted so far towards maintaining coherence at the expense of user intent, it may be a good idea to compensate by staying close to the operations.

Finally, if you decide to tilt towards intent preservation, there are definitely approaches to automate merges, avoiding nagging the user to fix conflicts. The most successful trivial solution remains latest-write-wins, which gives surprisingly good results assuming a high data granularity and a rich set of operations. But unless there is a way to automate coherence validation (which the equivalent in git projects is, I suppose, running the tests), we can only rely on user attention… in which case, relying on the user to fix coherence after the fact on a database that heavily preserves user intent would be pretty much the same.

So… do you plan on supporting custom coherence rules? Alternatively, which conflict resolutions are you leaning towards?

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#113

Post by mikergray » Wed, Aug 03, 2016, 12:06 AM UTC

shruubi wrote: post #103

I really like the idea in theory, but seeing it in practice I feel the whole thing is too concerned with being a wrapper around git handling for their dataset files. I would much rather see diffs based around the records themselves, and not so much the structure of the data.

While Git is referenced as an inspiration, the implementation of Noms does not use Git. Noms performs diffs on the data - as records or whatever other structure you used in importing your data. CSV is but one example of a way to import data into Noms, but since so much data is available in that format it is an easy one to reference that most people know. Noms can also import JSON, XML and many other data types if you are willing to write JS or Go code (more to come). Thanks for taking a look at Noms!

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#114

Post by carterehsmith » Wed, Aug 03, 2016, 12:20 AM UTC

im_down_w_otp wrote: post #104

fiatjaf wrote: post #99

Earlier quoted context omitted.

In case of conflicts, CouchDB assumes the most modified branch of the document (i.e., the document with the higher revision number) is the winner. You can resolve the conflict by choosing a different branch/revision manually, but you can also choose to not do anything.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

I think the OP meant "most recently changed", not "most changed" :)

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#115

Post by im_down_w_otp » Wed, Aug 03, 2016, 12:24 AM UTC

espadrine wrote: post #112

rafaelweinstein wrote: post #106

Earlier quoted context omitted.

This is a question that we've gotten quite a bit. It's our view that there's no magic solution to conflicts. There are logical conflicts in the real world that must be arbitrated. That said, it's a surprisingly basic thing, but just knowing what changed from party (a) and party (b)'s perspective (relative to their most recently agreed-upon state) is somewhat rare or ad-hoc in existing systems. In noms, you can direct…

To me, there are two separate concerns: intent preservation and coherence. Operations cause a change of state. The sequence of operations that are performed given a user action need to ensure that they will cause a modification of the state of the database in the way intended by the user. Coherence is about maintaining a design. A set of rules are conceived. For each state, it is possible to determine whether the sta…

What is "latest" in a situation where you're experiencing concurrent writes?

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#116

Post by aboodman » Wed, Aug 03, 2016, 12:27 AM UTC

zphds wrote: post #49

Going through the SDK docs, why was a scheme like ' http://localhost:8000::people' chosen instead of the plain old ' http://localhost:8000/people' ? Are there any benefits? If yes, curious to know what they are.

Thanks for the help everyone with this most important aspect of the system ;).

To clarify, we don't think of these specs as URLs. The part before the final double-colon is a URL. To parse one, you get the final double colon, and take everything to the left as a URL.

There's some info on the syntax here:

https://github.com/attic-labs/noms/blob/master/doc/spelling....

Though it's not presented as a formal grammar in that doc, our most important criteria for the syntax was:

  - unambiguousness
  - interacts well with the shell, since we frequently use these as part of command lines

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#117

Post by rafaelweinstein » Wed, Aug 03, 2016, 12:34 AM UTC

espadrine wrote: post #112

rafaelweinstein wrote: post #106

Earlier quoted context omitted.

This is a question that we've gotten quite a bit. It's our view that there's no magic solution to conflicts. There are logical conflicts in the real world that must be arbitrated. That said, it's a surprisingly basic thing, but just knowing what changed from party (a) and party (b)'s perspective (relative to their most recently agreed-upon state) is somewhat rare or ad-hoc in existing systems. In noms, you can direct…

To me, there are two separate concerns: intent preservation and coherence. Operations cause a change of state. The sequence of operations that are performed given a user action need to ensure that they will cause a modification of the state of the database in the way intended by the user. Coherence is about maintaining a design. A set of rules are conceived. For each state, it is possible to determine whether the sta…

Much of this is yet to be designed, but in general, our approach has been to lean towards a relatively layered system. That is to say, that at the lowest level, noms probably won't make a judgement about the trade-off you describe above, but rather provide primitives which allow layers above to more easily take opinionated positions.

In the near-term we'll be paying attention to specific cases that arise, and we'd welcome the opportunity to learn more about those that you may encounter.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#118

Post by im_down_w_otp » Wed, Aug 03, 2016, 12:34 AM UTC

carterehsmith wrote: post #114

im_down_w_otp wrote: post #104

Earlier quoted context omitted.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

I think the OP meant "most recently changed", not "most changed" :)

I don't think so. Though "most recently changed" is pretty useless too. They won't be synchronized in a distributed system, and even on a single machine if it's setup to use something like NTP, then the time won't be monotonically increasing since the clock-sync mechanism can move time both forward and backward.

Having now looked at it out of curiousity... http://guide.couchdb.org/draft/conflicts.html

"Each revision includes a list of previous revisions. The revision with the longest revision history list becomes the winning revision. If they are the same, the _rev values are compared in ASCII sort order, and the highest wins. So, in our example, 2-de0ea16f8621cbac506d23a0fbbde08a beats 2-7c971bb974251ae8541b8fe045964219."

Weird.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#119

Post by carterehsmith » Wed, Aug 03, 2016, 12:39 AM UTC

aboodman wrote: post #18

pinko wrote: post #8

Earlier quoted context omitted.

Another question to help me better understand the tool: who do you see as your competitors, technically? What do you see as viable alternatives to Noms, but worse? (Or better!) I do see that you were inspired by git, but clearly your use-cases are different.

Git is a competitor. It is fairly common to check data (e.g., csv or json files) into Git today. However, this falls down pretty rapidly. In order to get reasonable diffs, the data has to be sorted, and line-oriented. Also Git just doesn't scale well to larger repos or individual objects. Otherwise, we see the competitors as the way that people distribute data today - custom APIs, zip files full of CSV, etc.

> Also Git just doesn't scale well to larger repos or individual objects.

I guess the above means that Noms does scale to larger repos... Do you guys have any numbers, comparisons, benchmarks against git?

If so, it would be useful to include in the readme.md as it would be kind of a big thing, and quite attractive to many people.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#120

Post by espadrine » Wed, Aug 03, 2016, 12:40 AM UTC

im_down_w_otp wrote: post #115

espadrine wrote: post #112

Earlier quoted context omitted.

To me, there are two separate concerns: intent preservation and coherence. Operations cause a change of state. The sequence of operations that are performed given a user action need to ensure that they will cause a modification of the state of the database in the way intended by the user. Coherence is about maintaining a design. A set of rules are conceived. For each state, it is possible to determine whether the sta…

What is "latest" in a situation where you're experiencing concurrent writes?

Google's Spanner, for instance, relies on their TrueTime design, which requires having a GPS clock and an atomic clock on each datacenter, I believe. Most designs simply rely on NTP or a similar time synchronization system.

Another approach is to maintain total order of writes. Assuming some form of consensus protocol to determine write order, the unicity of the order ensures synchronization. That design, however, tends to preserve user intent less. Bitcoin has a form of that.