Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB 2.1 is out: high availability

rethinkdb.com

Post reply on HN
Print view View original

11–20 of 108 posts

Re: RethinkDB 2.1 is out: high availability

#11

Post by coffeemug »

ukd1 wrote: post #5
coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

Awesome work guys. Could you give some details on what you're most proud of and what was most challenging?

There are two parts to this that are really challenging -- designing a robust system (since there are a million different edge cases that can happen in a distributed system), and designing an intuitive interface so that the database does the right thing out of the box, is configurable if users want that, and is actually easy to configure in practice.

Doing each thing independently is hard; doing both of them together in one system is really really hard. I'm particularly proud of how nicely everything clicks together, and how elegant the administration API is. It seems very simple and is extremely powerful; getting that right took many iterations over the course of a year.

Re: RethinkDB 2.1 is out: high availability

#12

Post by karlgrz »

coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

Fantastic news, this looks great! Just a quick question, can I update individual nodes one at a time in my running cluster, or do they all need to be on 2.1 at the same time?

Re: RethinkDB 2.1 is out: high availability

#13

Post by danielmewes »

karlgrz wrote: post #12
coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

Fantastic news, this looks great! Just a quick question, can I update individual nodes one at a time in my running cluster, or do they all need to be on 2.1 at the same time?

Unfortunately 2.0 nodes cannot connect to 2.1 nodes and vice versa, so depending on your table configuration you might need to update all of them at the same time.

Re: RethinkDB 2.1 is out: high availability

#14

Post by mikegioia »

coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

I'm evaluating a move from CouchDB to another system purely as an easily sharded/replicated KV document store. Do you recommend Rethink for that specific use-case or is it overkill? Your docs seem to convey that this isn't really in Rethink's wheelhouse (in that RDB is more suited for querying or real-time updating).

Re: RethinkDB 2.1 is out: high availability

#15

Post by coffeemug »

mikegioia wrote: post #14
coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

I'm evaluating a move from CouchDB to another system purely as an easily sharded/replicated KV document store. Do you recommend Rethink for that specific use-case or is it overkill? Your docs seem to convey that this isn't really in Rethink's wheelhouse (in that RDB is more suited for querying or real-time updating).

RethinkDB can definitely act as a distributed key-value store (and a pretty good one). But I think the choice is highly dependent on your use-case and requirements. Could you go into a little more depth wrt your workload? (e.g. number of records, average record size, read/write ratio, expected ops/second, etc.)

Re: RethinkDB 2.1 is out: high availability

#16

Post by eloff »

coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

I'm curious about "We chose to only store cluster metadata in Raft due to performance limitations imposed by distributed consensus"

I would naively assume that since you need to replicate your database to the slave servers, you need to send the WAL or changes somehow anyway. Could you elaborate on what was the problem there? I'd be surprised if it is because of a fundamental limitation of Raft (but could easily be a limitation of a Raft library!)

Re: RethinkDB 2.1 is out: high availability

#17

Post by jchrisa »

mikegioia wrote: post #14
coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

I'm evaluating a move from CouchDB to another system purely as an easily sharded/replicated KV document store. Do you recommend Rethink for that specific use-case or is it overkill? Your docs seem to convey that this isn't really in Rethink's wheelhouse (in that RDB is more suited for querying or real-time updating).

If you like the general style of CouchDB but want scale and availability that's been proven in mission critical user facing deployments for years, we just did a blog post about moving to Couchbase http://blog.couchbase.com/2015/august/moving-couch

Re: RethinkDB 2.1 is out: high availability

#18

Post by coffeemug »

eloff wrote: post #16
coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

I'm curious about "We chose to only store cluster metadata in Raft due to performance limitations imposed by distributed consensus" I would naively assume that since you need to replicate your database to the slave servers, you need to send the WAL or changes somehow anyway. Could you elaborate on what was the problem there? I'd be surprised if it is because of a fundamental limitation of Raft (but could easily be a…

Suppose you configure RethinkDB to store three copies of the data. When you do a write, RethinkDB will replicate the data to three nodes; that operation is roughly a matter of sending three messages to the appropriate nodes, getting the acks, and sending the ack back to the client.

If we used Raft to replicate the document, in many cases it would be a lot more chatty. If you look at the Raft paper and track all the messages that would have to go back and forth, it would dramatically increase the latency for each write. This is inherent to Raft (and any distributed consensus protocol). Unfortunately distributed consensus isn't free; you have to pay pretty heavy latency costs which would be unacceptable in production, so we couldn't just uniformly apply that to every write.

Re: RethinkDB 2.1 is out: high availability

#19

Post by hipsterrific »

coffeemug wrote: post #2

Slava @ RethinkDB here. I'll be around all day to answer questions about the release (along with a few other engineers on our team). We're very excited about this release -- it makes the lives of RethinkDB users dramatically better because they won't have to wake up anymore in the middle of the night in case of most hardware failures :) It also took over a year to build and test, and has been one of the most challeng…

I'm excited to work with RethinkDB on my pet projects. Glad to hear you guys beefed up failovers, that's a big deal in a lot of places. ;)

Re: RethinkDB 2.1 is out: high availability

#20

Post by mikegioia »

coffeemug wrote: post #15
mikegioia wrote: post #14

Earlier quoted context omitted.

I'm evaluating a move from CouchDB to another system purely as an easily sharded/replicated KV document store. Do you recommend Rethink for that specific use-case or is it overkill? Your docs seem to convey that this isn't really in Rethink's wheelhouse (in that RDB is more suited for querying or real-time updating).

RethinkDB can definitely act as a distributed key-value store (and a pretty good one). But I think the choice is highly dependent on your use-case and requirements. Could you go into a little more depth wrt your workload? (e.g. number of records, average record size, read/write ratio, expected ops/second, etc.)

Yea of course. We're basically talking about 10/90 write/read on JSON documents ~100kb in size with very low load, probably no more than 10-20 ops/second at peak. Records now are at 200,000 projected to grow 250k/year.
Post reply on HN