Does YugaByte still use the Raft and HybridTime implementations from Apache Kudu? If so, how relevant are these results for Kudu?
[deleted]
Jepsen: YugaByte DB 1.1.9
41–49 of 49 posts
Re: Jepsen: YugaByte DB 1.1.9
#42Does YugaByte still use the Raft and HybridTime implementations from Apache Kudu? If so, how relevant are these results for Kudu?
[deleted]
Re: Jepsen: YugaByte DB 1.1.9
#43Does YugaByte still use the Raft and HybridTime implementations from Apache Kudu? If so, how relevant are these results for Kudu?
I wanted to add a few details to the previous reply. While the Raft/HybridTime implementation has its roots in Apache Kudu the results will NOT be quite applicable to Kudu. Aside from the fact that the code base has evolved/diverged over the 3+ years, there are key/relevant areas (ones very relevant to these Jepsen tests) where YugaByte DB has added capabilities or follows a different design than Kudu. For example: -…
FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 (https://github.com/apache/kudu/commit/535dae) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.
Re: Jepsen: YugaByte DB 1.1.9
#44Earlier quoted context omitted.
I wanted to add a few details to the previous reply. While the Raft/HybridTime implementation has its roots in Apache Kudu the results will NOT be quite applicable to Kudu. Aside from the fact that the code base has evolved/diverged over the 3+ years, there are key/relevant areas (ones very relevant to these Jepsen tests) where YugaByte DB has added capabilities or follows a different design than Kudu. For example: -…
FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 ( https://github.com/apache/kudu/commit/535dae ) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.
hi @mpercy,
Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability).
- To make the "online" piece of the membership change work correctly we added support for LEARNER (PRE VOTER) role (where the new member enters in a non-voting mode till it's caught up). https://github.com/YugaByte/yugabyte-db/commit/909d26e31ecd0....
- Load Balancing (which uses the membership changes) is automatic. (https://github.com/YugaByte/yugabyte-db/commit/e4667eb7ec0e6...)
- Remote bootstrap (due to membership changes) also has undergone substantial changes given that YugaByte DB uses a customize/extended version of RocksDB as the storage engine and does a tighter coupling of Raft with RocksDB storage engine. (https://github.com/YugaByte/yugabyte-db/blob/master/docs/ext...)
- Dynamic Leader Balancing is also new-- it causes leadership to be proactively altered in a running system to ensure each node is the leader for a similar number of tablets.
regards, Kannan
Re: Jepsen: YugaByte DB 1.1.9
#45Earlier quoted context omitted.
FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 ( https://github.com/apache/kudu/commit/535dae ) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.
hi @mpercy, Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability). - To make the "online" piece of the membership change work correctly we added support for LEARNER (PRE VOTER) role (where the new member enters in a non-voting mode till it's caught up). https://github.com/YugaBy…
Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing (https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.h...).
I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern.
I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus.
By the way, we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285
We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.
Re: Jepsen: YugaByte DB 1.1.9
#46Earlier quoted context omitted.
hi @mpercy, Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability). - To make the "online" piece of the membership change work correctly we added support for LEARNER (PRE VOTER) role (where the new member enters in a non-voting mode till it's caught up). https://github.com/YugaBy…
Interesting. Just last year we implemented improved re-replication ( https://github.com/apache/kudu/commit/79a255 ) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing ( https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.h... ). I'm curious if you did anything to prevent automatic rebalancing from being trig…
Yes - load-balancing is automatic and turned on by default. Two key protections we did there were:
- A per-node rate limit for sending/receiving on behalf of remote bootstrap (https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5dd...)
- A global throttle on how many quorum membership changes are in flight cluster wide.
On the CDC question - we added an OBSERVER role as well (a non-voting quorum member) to support for features like read-replicas. This is currently not exposed via an external API - but CDC is something we are working on now.
Happy to exchange notes with your team!
regards, Kannan
Re: Jepsen: YugaByte DB 1.1.9
#47Earlier quoted context omitted.
hi @mpercy, Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability). - To make the "online" piece of the membership change work correctly we added support for LEARNER (PRE VOTER) role (where the new member enters in a non-voting mode till it's caught up). https://github.com/YugaBy…
Interesting. Just last year we implemented improved re-replication ( https://github.com/apache/kudu/commit/79a255 ) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing ( https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.h... ). I'm curious if you did anything to prevent automatic rebalancing from being trig…
Forgot to respond to your:
https://github.com/apache/kudu/commit/ebb285 >>
Nice! We too ended up adding similar support for multi-AZ/multi-region aware replication configurations starting here (https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934fa...)
regards, Kannan
Re: Jepsen: YugaByte DB 1.1.9
#48Earlier quoted context omitted.
We use cockroachDB in production and before that we were on MySQL and as of yet we don’t have a specific usecase where we use serializable transactions. Snapshot isolation or even read committed is just fine. So I don’t think it’s absolutely necessary
To be clear there’s no way around serializable transactions in cockroachDB. We have had to adapt our monolith to it (we’re thinking of ways to make it more nimble by breaking out services etc). But the point I was making was that we had MySQL for a while and never ran into issues with its isolation levels until it stopped scaling. Instead of vitess or some other MySQL system we went with cockroach after finding vites…
First time I've heard about a production use case. Care to share any details?
Re: Jepsen: YugaByte DB 1.1.9
#49First time I heard about them. Seems to be another distributed SQL (aka 'newsql') alternative to TiDB and CockroachDB. Based on RocksDB (like Cockroach) with a custom distributed key/val layer and and additional SQL layer on top. PostgreSQL protocol compatible. OS with Apache license. Seems interesting. (when ignoring the "planet scale SQL" marketing speak... [1]) [1] https://www.yugabyte.com/planet-scale-sql/
As an expert in the DB space, I'm extraordinarily cynical. But their willingness to license it as truly open, Apache-style, instantly is a big win-over. I'm a competitor, but I can tell these guys/gals are genuine in their efforts. We need more people, teams, and DBs like YugaByteDB in the world. Thank you for your efforts.
Thanks for your honest insight man, really appreciate it! Gun is also extremely cool, here's the link for others https://github.com/amark/gun