Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serializable, Lockless, Distributed: Isolation in CockroachDB

cockroachlabs.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#31

Post by LoSboccacc »

ansible wrote: post #29
LoSboccacc wrote: post #25

By coincidence I was looking around for a datastore. I almost settled on couchbase when I saw this; easy cluster deployment is my main draw after looking at how bad setting up master-master system is in other solutions (hbase, mongo, couchdb all have eterogeneus nodes and weird failure modes) One quick question since I saw the devs around and I can't find a final answer on it on google: are there any strong roadblock…

I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distributed blob store, and just store the index in the distributed database. Of course, now you are half way towards a distributed filesystem...

but then you'd need to store file path in the datastore, with all the failure modes that follows.

I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again

I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning experience, but since it's nowhere close to being workable I am also looking around for what's happening in this space.

I've this longer term goal about having a serverless framework where the datastore also provide oauth for users, which was inspired by https://github.com/serverless/serverless - would be a boon for client side webapp app style startups.

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#32

Post by manigandham »

LoSboccacc wrote: post #31
ansible wrote: post #29

Earlier quoted context omitted.

I don't know that you would be happy trying to store blobs of that size in any distributed database, current or future. I'd suggest having an immutable distributed blob store, and just store the index in the distributed database. Of course, now you are half way towards a distributed filesystem...

but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning exp…

I have a strong sense that you're vastly over-engineering something.

The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability.

Storing a tiny bit of text for the filename is trivial and you can use any durable database for that.

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#33

Post by manigandham »

chillaxtian wrote: post #3
tyingq wrote: post #2

I love the transparency this project operates with. Looking forward to when joins are implemented so I can try using it with something non-trivial.

> Looking forward to when joins are implemented so I can try using it with something non-trivial. these types of databases typically don't have joins. you'll be waiting a while.

MemSQL does distributed joins really well, using a combination of in-memory rowstore + disk columnstore and the ability to have certain tables replicated to every node for faster joining.

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#34

Post by LoSboccacc »

manigandham wrote: post #32
LoSboccacc wrote: post #31

Earlier quoted context omitted.

but then you'd need to store file path in the datastore, with all the failure modes that follows. I know what the current best practices are, but for bootstrapped one man startups you always end up with dozen moving parts to be tamed and instead of building stuff you need to solve the same problems over and over again I'm implementing a datastore on top of glusterfs, which while amateurish is a wonderful learning exp…

I have a strong sense that you're vastly over-engineering something. The solved problem is called an object store. Just use S3 or Azure Storage or Google Cloud Storage - they all have options for geo-replication and it'll be just as fast or faster than any database replication you set up, while being far more 9's of availability and reliability. Storing a tiny bit of text for the filename is trivial and you can use a…

> I have a strong sense that you're vastly over-engineering something.

and you're 100% right it wouldn't be interesting otherwise :D (I'm not gonna use this on real work no worries)

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#35

Post by tyingq »

biokoda wrote: post #23
andrewvc wrote: post #13

Earlier quoted context omitted.

Joins in distributed databases all must make some sort of unsavory tradeoff be it speed or space or limiting what you can join. So, while any distributed database can do a join, it may not be fast or flexible enough to be worth it.

Joins in ActorDB work great. Of course that is because we use an entirely different way of making an SQL database distributed and our joins aren't actually distributed even though the database is. Best way to solve a problem is to avoid it.

I looked at ActorDB, with the thought of using it as a sort of "Sql Enabled" version of etcd. Meaning, storing config data, mostly reads, with no big performance requirements.

But, the use case is to ensure that the config data is available on all nodes....high availability. So, for example, sharding isn't wanted or needed.

It was difficult, however, to get my arms around the whole "actor model", and understand how to use ActorDB in this relatively simple deployment.

Long story short, is there some reference, or dead simple example of a "single actor" deployment with a small number of tables?

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#36

Post by smoyer »

LoSboccacc wrote: post #25

By coincidence I was looking around for a datastore. I almost settled on couchbase when I saw this; easy cluster deployment is my main draw after looking at how bad setting up master-master system is in other solutions (hbase, mongo, couchdb all have eterogeneus nodes and weird failure modes) One quick question since I saw the devs around and I can't find a final answer on it on google: are there any strong roadblock…

As a long-time CouchDB user, I'm going to assume that by "weird failure modes", you mean "conflicts". There are some awesome aspects to using CouchDB, but getting used to conflicts certainly took some time. In short, your application needs to include the code to resolve problems that arise due to network partitions or concurrent writes to different nodes. On the other hand, it's a really stable platform and the write-only database files worked really well (we lost zero data in over 5 years of operation ... even with tests that included powering down a node while in use).

But ... I also believe that CockroachDB is going to be the project to finally solve these problems and it's got an SQL interface! I'm waiting for SQL joins, and then I suspect I'll be moving at least some of my clustered PostgreSQL instances to CockroachDB.

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#37

Post by sriram_malhar »

teraflop wrote: post #14
chillaxtian wrote: post #3

Earlier quoted context omitted.

> Looking forward to when joins are implemented so I can try using it with something non-trivial. these types of databases typically don't have joins. you'll be waiting a while.

CockroachDB doesn't currently support joins, but it's been designed so that it can potentially support them, unlike most NoSQL databases. In particular, it supports cross-machine transactions, which are a critical building block for both correctness and performance. For instance, suppose you want to join on a column that's not a primary key. No big deal in a typical RDBMS; just do an index lookup on the join column.…

Yes, distributed joins are well understood. But I'm not at all convinced that the relational model is a good fit for a scale-out, widely distributed, fault-tolerant database. The engineering decisions for each of these aspects add enough latency to the process as it is.

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#38

Post by ams6110 »

teraflop wrote: post #26
ams6110 wrote: post #22

Earlier quoted context omitted.

I think it does require locking, because in PostGres (or Oracle) readers do not block writers and writers do not block readers. So to be sure you update the same version you read, you have to select...for update.

Having serializable transactions is equivalent to adding "FOR UPDATE" to every SELECT statement, so it sounds like CockroachDB already does what you want. A typical RDBMS will prevent conflicts by forcing queries to block until they can be executed in a conflict-free ordering. CockroachDB instead detects conflicts after the fact and prevents inconsistent transactions from committing, forcing them to retry. The end re…

GP said that the use case does not require locking, but in PgSQL (which was mentioned) or Oracle, it does. The default transaction isolation level is not serializable. You don't read uncommitted updates, but reads are not repeatable unless you explicitly ask for that. If you do something like this (in a transaction):

   select ... from T where ;
   ...
   update T ... where ;
   commit;
there is no guarantee that the row you are updating is the same as the one you selected, unless you add "for update" to the select.

Also note that two successive SELECT commands can see different data, even though they are within a single transaction, if other transactions commit changes after the first SELECT starts and before the second SELECT starts.

-- http://www.postgresql.org/docs/current/static/transaction-is...

A query acquires no data locks. Therefore, other transactions can query and update a table being queried, including the specific rows being queried. Because queries lacking FOR UPDATE clauses do not acquire any data locks to block other operations, such queries are often referred to in Oracle as nonblocking queries.

-- http://docs.oracle.com/cd/B19306_01/server.102/b14220/consis...

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#39

Post by sriram_malhar »

danieltoshea wrote: post #6

How does this work if the clocks drift between the nodes? Does this allow incorrect behavior because one transaction looks like it happened before another?

For serializability, all you care about it is some sequential order. You get that with hybrid logical clocks (http://www.cse.buffalo.edu/tech-reports/2014-04.pdf). It gives you a monotonically increasing timestamp that you can use instead of dumb version numbers.

On the other hand, if you want to ensure linearizability, you do care about the worst case clock drift, which in CockroachDB is a configurable parameter. One can adopt Google's Spanner's approach ("commit wait"), which is to wait out the response to the client to ride out NTP uncertainty (typically a few milliseconds inside a data center, but 100s of milliseconds in the wide area).

Re: Serializable, Lockless, Distributed: Isolation in CockroachDB

#40

Post by teraflop »

ams6110 wrote: post #38
teraflop wrote: post #26

Earlier quoted context omitted.

Having serializable transactions is equivalent to adding "FOR UPDATE" to every SELECT statement, so it sounds like CockroachDB already does what you want. A typical RDBMS will prevent conflicts by forcing queries to block until they can be executed in a conflict-free ordering. CockroachDB instead detects conflicts after the fact and prevents inconsistent transactions from committing, forcing them to retry. The end re…

GP said that the use case does not require locking, but in PgSQL (which was mentioned) or Oracle, it does. The default transaction isolation level is not serializable. You don't read uncommitted updates, but reads are not repeatable unless you explicitly ask for that. If you do something like this (in a transaction): select ... from T where ; ... update T ... where ; commit; there is no guarantee that the row you are…

Sorry, I don't understand this comment because I can't tell if you're disagreeing with me about anything.

> GP said that the use case does not require locking, but in PgSQL (which was mentioned) or Oracle, it does.

Right, it does in a typical RDBMS, but not in CockroachDB. The definition of an isolation level is defined in terms of what interactions are possible between concurrent successful transactions. Locks, or the lack of locks, are an implementation detail.

> The default transaction isolation level is not serializable. ... Also note that two successive SELECT commands can see different data, even though they are within a single transaction, if other transactions commit changes after the first SELECT starts and before the second SELECT starts.

I agree. If you set the isolation level to "serializable", such anomalies aren't possible, even if you don't use FOR UPDATE.

Post reply on HN