Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Noms – A new decentralized database based on ideas from Git

medium.com

Post reply on HN
Print view View original

121–130 of 180 posts

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#121

Post by zphds »

aboodman wrote: post #116
zphds wrote: post #49

Going through the SDK docs, why was a scheme like ' http://localhost:8000::people' chosen instead of the plain old ' http://localhost:8000/people' ? Are there any benefits? If yes, curious to know what they are.

Thanks for the help everyone with this most important aspect of the system ;). To clarify, we don't think of these specs as URLs. The part before the final double-colon is a URL. To parse one, you get the final double colon, and take everything to the left as a URL. There's some info on the syntax here: https://github.com/attic-labs/noms/blob/master/doc/spelling.... Though it's not presented as a formal grammar in th…

> To clarify, we don't think of these specs as URLs

That makes it a lot clearer. :). Looking forward to take noms for a spin soon.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#122

Post by batbomb »

aboodman wrote: post #116
zphds wrote: post #49

Going through the SDK docs, why was a scheme like ' http://localhost:8000::people' chosen instead of the plain old ' http://localhost:8000/people' ? Are there any benefits? If yes, curious to know what they are.

Thanks for the help everyone with this most important aspect of the system ;). To clarify, we don't think of these specs as URLs. The part before the final double-colon is a URL. To parse one, you get the final double colon, and take everything to the left as a URL. There's some info on the syntax here: https://github.com/attic-labs/noms/blob/master/doc/spelling.... Though it's not presented as a formal grammar in th…

> To clarify, we don't think of these specs as URLs.

But everyone else will because you are including the protocol, and at the end of the day, they are a uniform way of identifying a resource, so they are functionally URIs.

Otherwise, you should probably either conform to the HTTP(S) protocol spec or makeup your own, e.g. noms+http://dbinstance.noms.foo::database/dataset

SQLAlchemy and most DB URIs are good examples on how to do this. For example, you can connect to a MySQL database instance and give it a default namespace/schema/database.

Part of the issue here is the ambiguity between a database, a database instance/server/host, a dataset/table, a catalog/namespace/schema, and what all those words and concepts mean. There's little consensus across fields, because even if computer scientists say "Okay, this is what a dataset actually is", somebody, whether it's a biologist or a physicist, will throw up their arms in protest.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#123

Post by im_down_w_otp »

espadrine wrote: post #120
im_down_w_otp wrote: post #115

Earlier quoted context omitted.

What is "latest" in a situation where you're experiencing concurrent writes?

Google's Spanner, for instance, relies on their TrueTime design, which requires having a GPS clock and an atomic clock on each datacenter, I believe. Most designs simply rely on NTP or a similar time synchronization system. Another approach is to maintain total order of writes. Assuming some form of consensus protocol to determine write order, the unicity of the order ensures synchronization. That design, however, te…

Time isn't a reliable resource in this context.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#124

Post by [deleted] »

bkalman wrote: post #74
zphds wrote: post #70

Earlier quoted context omitted.

But isn't ` / ` more or less similar to ` :: `? The only difference is the choice of a delimiter to disambiguate between a database and a dataset. For me, the first scheme is much more familiar.

Say we did just do / . What does the path " http://demo.noms.io/cli-tour/sf-fire-inspections/raw" refer to? Is the database " http://demo.noms.io" and the dataset "cli-tour/sf-fire-inspections/raw"? Is the database " http://demo.noms.io/cli-tour/sf-fire-inspections" and the dataset "raw"? In our sample data (see https://github.com/attic-labs/noms/blob/master/doc/cli-tour.... for example) we actually have this exact p…

[deleted]

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#125

Post by adambrenecki »

bkalman wrote: post #74
zphds wrote: post #70

Earlier quoted context omitted.

But isn't ` / ` more or less similar to ` :: `? The only difference is the choice of a delimiter to disambiguate between a database and a dataset. For me, the first scheme is much more familiar.

Say we did just do / . What does the path " http://demo.noms.io/cli-tour/sf-fire-inspections/raw" refer to? Is the database " http://demo.noms.io" and the dataset "cli-tour/sf-fire-inspections/raw"? Is the database " http://demo.noms.io/cli-tour/sf-fire-inspections" and the dataset "raw"? In our sample data (see https://github.com/attic-labs/noms/blob/master/doc/cli-tour.... for example) we actually have this exact p…

Why not have the dataset name as a fragment in the URL? For instance:

    http://demo.noms.io/cli-tour#sf-fire-inspections/raw
Glancing over RFC3986 [1], fragment identifiers seem to be pretty much made for what you're trying to communicate with :: - separating a subresource (the dataset) from a primary resource (the database). Unless I'm misunderstanding something?

[1]: https://tools.ietf.org/html/rfc3986#section-3.5

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#126

Post by aboodman »

alexatkeplar wrote: post #76
aboodman wrote: post #2

Hi all. I'm one of the creators of Noms. Happy to answer any questions!

Why would you exclusively support schema inference, rather than also allowing users to manually specify their schemas? Schema inference is very difficult to do correctly and safely, especially with small initial samples of instances (source: work on https://github.com/snowplow/schema-guru ).

There's a little bit of terminology overloading going on here.

In Noms every value has a type. It's an immutable system, so this type just is. The type of `42` is `Number`. The type of `"foobar"` is `String`. The type of `[42,44]` is `List`. And if you add "foo" to that list, the type becomes `List`.

We don't try to infer a general database schema from a few instances of data. We just apply this aggregation up the tree and report the result.

That all said, we do want to eventually add schema _validation_, by which I mean the ability to associate a type with a dataset and have the database enforce that any value committed to the dataset is compatible with that type (following subtyping rules).

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#127

Post by aboodman »

amelius wrote: post #60
aboodman wrote: post #2

Hi all. I'm one of the creators of Noms. Happy to answer any questions!

How does Noms compare to ipfs? https://ipfs.io/

First, IPFS is awesome.

IPFS is essentially providing a globally decentralized filesystem. Noms is providing (or hopes to provide) a database.

By database, I mean:

  - small individual records
  - efficient queries, updates, and range scans
  - ability to support complex queries
  - ability to enforce structural data validity
These are all things that IPFS could eventually grow to support, but in order to do it, I think it would have to grow into or layer something like noms on top.

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#128

Post by rdtsc »

fiatjaf wrote: post #99
svckr wrote: post #95

Earlier quoted context omitted.

> [...] forcing the end-user to manually deal with fixing problems, which is often fraught with pain and peril, and doesn't seem like a strategy that would work for something that's a database (as opposed to something that's a workflow). Isn't that what (for example) CouchDB does? I believe the reasoning is that conflict resolution is often application specific, so why not deal with it in the application?

In case of conflicts, CouchDB assumes the most modified branch of the document (i.e., the document with the higher revision number) is the winner. You can resolve the conflict by choosing a different branch/revision manually, but you can also choose to not do anything.

Yes, it picks a winner, which it show on all machines (so all machines that have seen same changes will pick the same winner). But it also keeps conflicts around, so users who care about them can correctly resolve them.

Sometimes the winner it picks is not what the users want, That could surprising, but it is correct because it really is a user-level conflicts.

(Now, user may very well at a timestamp field to the document, hope ntp works well and resolve the conflicts if they appear based on that, but CouchDB tries not to make such assumption on behalf of the user).

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#129

Post by fiatjaf »

carterehsmith wrote: post #114
im_down_w_otp wrote: post #104

Earlier quoted context omitted.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

I think the OP meant "most recently changed", not "most changed" :)

No, I meant "most changed".

I don't see how this could be better for a deterministic approach. The recommendations are always that the developer must implement a saner way to resolve the conflicts.

In the CouchDB world, however, I have the impression that conflict resolution is ignored most of times, so we are left with this.

(I say this based on what I do, other people's code I read on the internet and the concerns of the CouchDB core developers about educating users and developers to setup saner conflict resolution approaches themselves.)

Re: Show HN: Noms – A new decentralized database based on ideas from Git

#130

Post by rdtsc »

im_down_w_otp wrote: post #104
fiatjaf wrote: post #99

Earlier quoted context omitted.

In case of conflicts, CouchDB assumes the most modified branch of the document (i.e., the document with the higher revision number) is the winner. You can resolve the conflict by choosing a different branch/revision manually, but you can also choose to not do anything.

Really? That's nearly undifferentiated from just picking one at random. How is, "whoever hits the queue most often" a useful deterministic resolution strategy? I mean I guess it's functionally no worse than wall-clock time or something, but still kinda funny. :-)

It is not random. It is consitently picking the same document on all servers that have seen the same changes. By default it picks the one with the most changes. That consintency ("the same" part is very important) it means if you replicate and bring in some conflicts, both sides will show the same state. So you won't randomly after replicating A to B, and B to A see document 1 as the winner on A but 2 on B. They'll both pick 1 or 2. So both would settle on the same state.

Also it doesn't delete or remove conflicting siblings, it is very good about not doing that to user data. Users only know exactly how to solve particular conflicts.

Post reply on HN