Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GoshawkDB: A distributed, transactional, fault-tolerant object store

goshawkdb.io

Post reply on HN
Print view View original

31–40 of 46 posts

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#31

Post by msackman »

helloiamaperson wrote: post #9
ctlaltdefeat wrote: post #4

This is really cool! I guess the main competition in terms of similar purpose and guarantees is http://hyperdex.org/ , although the add-on that enables transactions (quite critical...) is commercial.

The most production ready versions of this kind of thing are http://geode.incubator.apache.org and https://ignite.apache.org .

Having just had a quick look at both, the only point I'm willing to make is that ignite's distributed transactions use 2PC. https://apacheignite.readme.io/docs/transactions

2PC, at least in its usual forms, is not fool-proof: if failures happen in the "right" order then the whole txn gets blocked: it's actually proven to be equivalent to Paxos with F=0 (i.e. no ability to tolerate failure). On this point alone, GoshawkDB stands apart from anything that uses 2PC.

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#32

Post by dozzie »

msackman wrote: post #30
dozzie wrote: post #29

Earlier quoted context omitted.

> With GoshawkDB it really is up to the client as to what API you offer. So GoshawkDB is not an object database and apparently never will . That's OK, but don't call it object store. It's misleading, as it uses already-established term for something different, which also already has a name.

So that I understand, what is the definition of object store?

https://en.wikipedia.org/wiki/Object_database

Mind you, in several programming languages objects can have instance-specific methods (Ruby and Python being notable examples). This alone makes object more than merely a sum of a predefined structure having data fields and (also predefined) schema having functions to operate on those fields.

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#33

Post by msackman »

dozzie wrote: post #32
msackman wrote: post #30

Earlier quoted context omitted.

So that I understand, what is the definition of object store?

https://en.wikipedia.org/wiki/Object_database Mind you, in several programming languages objects can have instance-specific methods (Ruby and Python being notable examples). This alone makes object more than merely a sum of a predefined structure having data fields and (also predefined) schema having functions to operate on those fields.

Thanks for the link. Whilst I'm not arguing with your point, I believe I've never used the term "object database" to describe GoshawkDB, only "object store". I guess I'm struggling to find a more accurate term than "object store" to describe GoshawkDB. I don't like "document store", as to me "document" tends to be something human readable.

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#34

Post by dozzie »

msackman wrote: post #33
dozzie wrote: post #32

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Object_database Mind you, in several programming languages objects can have instance-specific methods (Ruby and Python being notable examples). This alone makes object more than merely a sum of a predefined structure having data fields and (also predefined) schema having functions to operate on those fields.

Thanks for the link. Whilst I'm not arguing with your point, I believe I've never used the term "object database" to describe GoshawkDB, only "object store". I guess I'm struggling to find a more accurate term than "object store" to describe GoshawkDB. I don't like "document store", as to me "document" tends to be something human readable.

I understand your reluctance for using term "document". I don't like it much either when it comes to JSON entities, but I can't find any better term, much less a widely used one.

Look at the matter this way: CouchDB, MongoDB, and ElasticSearch, all use the term "document" to describe hash containing data of JSON-compatible model. They already do so, and they are quite well known in the field. It only makes sense to follow their lead, so your potential users can recognize capabilities of your application easier.

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#35

Post by msackman »

dozzie wrote: post #34
msackman wrote: post #33

Earlier quoted context omitted.

Thanks for the link. Whilst I'm not arguing with your point, I believe I've never used the term "object database" to describe GoshawkDB, only "object store". I guess I'm struggling to find a more accurate term than "object store" to describe GoshawkDB. I don't like "document store", as to me "document" tends to be something human readable.

I understand your reluctance for using term "document". I don't like it much either when it comes to JSON entities, but I can't find any better term, much less a widely used one. Look at the matter this way: CouchDB, MongoDB, and ElasticSearch, all use the term "document" to describe hash containing data of JSON-compatible model. They already do so, and they are quite well known in the field. It only makes sense to f…

Excellent points well made :) I think I may well change the description then as you suggest, and once I have clients that support the same API model as things like ZODB then I'd describe it as something like "document store that can also be accessed like an object store" (well, hopefully not that long winded...).

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#36

Post by rystsov »

On the site it is written that GoshawkDB doesn't support membership change. Since you system is based on single decree paxos then you can take Raft's joint consensus and apply to plain paxos. Probably it's the simplest and easiest to change membership in paxos cluster.

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#37

Post by msackman »

rystsov wrote: post #36

On the site it is written that GoshawkDB doesn't support membership change. Since you system is based on single decree paxos then you can take Raft's joint consensus and apply to plain paxos. Probably it's the simplest and easiest to change membership in paxos cluster.

I don't think that's the problem though. The problem that I'm thinking of is that when a cluster grows in size, due to the use of consistent hashing, there'll be a set of objects that need to move between nodes. Calculating and achieving that movement is what concerns me. The exact properties are explained early on in http://arxiv.org/abs/1503.04988

I'm not expecting to ever need to model a global property of "these are the set of nodes that are up and running". I always worry about the effect of weird and wacky network issues on such systems.

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#39

Post by marknadal »

Hey msackman,

Pretty cool looking stuff. Where'd you get the design for object/graph structure? It is what we are using for our database, http://gunDB.io/ . Although we take the opposite approach to CAP than you, and have no support for serializability (I don't think this belongs in the protocol but instead in the data, like git).

What's you're take on concurrency? How do you handle conflicts (I assume this is easy for you with a master)?

Would love to jump on a chat too, being DB guys. Shoot me an email mark@gunDB.io welcome arounds!

Re: GoshawkDB: A distributed, transactional, fault-tolerant object store

#40

Post by rystsov »

msackman wrote: post #37
rystsov wrote: post #36

On the site it is written that GoshawkDB doesn't support membership change. Since you system is based on single decree paxos then you can take Raft's joint consensus and apply to plain paxos. Probably it's the simplest and easiest to change membership in paxos cluster.

I don't think that's the problem though. The problem that I'm thinking of is that when a cluster grows in size, due to the use of consistent hashing, there'll be a set of objects that need to move between nodes. Calculating and achieving that movement is what concerns me. The exact properties are explained early on in http://arxiv.org/abs/1503.04988 I'm not expecting to ever need to model a global property of "these…

By the way, I always was wondering why the schemas with consistent hashing maintained via the ring are more popular approach than treating key space as a line (-∞,+∞), maintaining the explicit map from segments and rays of keys to the replica groups and split the group/segment when replica group becomes too hot or heavy.

IMHO the perfect hashing solves the problem of distributing the data but the load may follow completely different patterns. See theInstagram's Justin Bieber problem.

Post reply on HN