[deleted]
Open-sourcing a 10x reduction in Apache Cassandra tail latency
31–40 of 171 posts
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#32Weird, did they try to use https://www.scylladb.com/ ?
I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.
There are those who've deployed on Java with tight latency requirements:
https://martinfowler.com/articles/lmax.html?t=1319912579 - Benchmarked at around 6 million transactions/second.
The issue isn't so much Java the language, as it is being aware of the GC, and developing with it in mind.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#33In a similar situation we just adjust the GC and started to use G1GC which resulted in similar numbers.
I bet that didn't take N engineers 12 months to build out, either
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#34We do want to contribute our work back to the Cassandra upstream, instead of keeping it as a fork. So that more users from C* community can benefit from the improvements. The pluggable storage engine is an ambitious project (https://issues.apache.org/jira/browse/CASSANDRA-13474). Any help will be appreciated!
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#35RocksDB is used all over Facebook, powers the entire social graph. Great storage engine that pairs well with multiple DBMS: MySQL, Mongo, Cassandra... We'll be at Percona Live 2018 in April, giving several talks, and are looking forward to hanging out and talking with users in our lounge area. We're working hard to support our open source community as well! https://github.com/facebook/rocksdb
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#36I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…
The purpose of separating into young and old generation is that it's easier to find dead objects in the young generation (as you said, average object lives for a short period of time). You only have to scan this subset for a minor GC. It doesn't really matter how many long-lived objects you have as long as you can avoid needing to do a major GC.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#37Nicely done! Looking forward to the pluggable storage engine.
The JIRA tickets don't really shine with much hope :/ https://issues.apache.org/jira/browse/CASSANDRA-13474 [2 comments from 2017 Apr] https://issues.apache.org/jira/browse/CASSANDRA-13475 [~100 comments, but the last one is from 2017 Nov, by the InstaG engineer] And the Rocksandra fork is already ~3500 commits behind master, so upstreaming this will be interesting. Oh, and the Rocksandra fork is already kind of aban…
I'm a committer, I'm familiar with the JIRA ticket.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#38Nicely done! Looking forward to the pluggable storage engine.
The JIRA tickets don't really shine with much hope :/ https://issues.apache.org/jira/browse/CASSANDRA-13474 [2 comments from 2017 Apr] https://issues.apache.org/jira/browse/CASSANDRA-13475 [~100 comments, but the last one is from 2017 Nov, by the InstaG engineer] And the Rocksandra fork is already ~3500 commits behind master, so upstreaming this will be interesting. Oh, and the Rocksandra fork is already kind of aban…
This is the rocksandra branch, https://github.com/Instagram/cassandra/tree/rocks_3.0, we develop it on top of Cassandra 3.0. It's the code we are running on our production servers.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#39I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…
This is correct; the standard approach here is to use regular c-style memory management for the data the system is managing, and the JVM heap only for the database "infrastructure".
This hybrid approach gives the benefit of a managed runtime and safety of GC for most of your code, but allows the performance of raw pointers/malloc for key code paths.
Some examples of this pattern on the JVM:
- The Neo4j Page Cache, Muninn, https://github.com/neo4j/neo4j/blob/3.4/community/io/src/mai...
- The Netty projects implementation of jemalloc for the JVM: https://github.com/netty/netty/blob/4.1/buffer/src/main/java...
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#40I remember using quite early versions of Cassandra back in an ad-tech startup I was at back in 2009 or 2010, spending unfortunate amounts of time fighting the JVM GC and trying to tune things so it behaved responsibly. It was a real problem then and I know a lot of work went into fixing GC behaviour. Then I stopped using Cassandra for work, but it's unfortunate this is still an issue?
What I took out of that is that I really feel like something like Cassandra is better suited to implementation in a language like C++ or Rust. And I believe others have since come along and done this.
I really liked the gossip-based federation in Cassandra though.