61–70 of 171 posts
I wonder how the numbers would have looked with the new low latency GC for Hotspot (ZGC). https://wiki.openjdk.java.net/display/zgc/Main
Early results from SPECjbb2015 are impressive. https://youtu.be/tShc0dyFtgw?t=5m1s
Earlier quoted context omitted.
Does FB still use Cassandra? I thought they abandoned them ages ago and then databricks picked it up?
The article is literally written by Instagram, which is FB.
Earlier quoted context omitted.
The purpose of separating into young and old generation is that it's easier to find dead objects in the young generation (as you said, average object lives for a short period of time). You only have to scan this subset for a minor GC. It doesn't really matter how many long-lived objects you have as long as you can avoid needing to do a major GC.
Don't you still need to scan the old generation during minor GC, in case a field in one of the older objects was modified to point to an object in the young generation? Or are there optimizations you can use to quickly and efficiently find references from the older generation to the younger?
This is typically handled by https://en.wikipedia.org/wiki/Write_barrier#In_Garbage_colle...
I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…
This is correct; the standard approach here is to use regular c-style memory management for the data the system is managing, and the JVM heap only for the database "infrastructure". This hybrid approach gives the benefit of a managed runtime and safety of GC for most of your code, but allows the performance of raw pointers/malloc for key code paths. Some examples of this pattern on the JVM: - The Neo4j Page Cache, Mu…
I wonder how long before we see a similar result from ElasticSearch. (Only other huge JVM based store I can think of).
Earlier quoted context omitted.
That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.
Even in a database, most of the code isn't performance sensitive. Making your life a bit harder in the fast path so it's easier in the slow path is at least a tradeoff worth considering.
So if you're latency sensitive then all of your code needs to be aggressive at avoiding object creation. All of your code becomes part of "the fast path", even if it's in a different thread.
Or you isolate your fast path in a different process or a non-GC'd runtime, the later being the approach taken here by Instagram.
No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your throughput (by 2.5%), just like any other thing consuming CPU cycles.
> We defined a metric called GC stall percentage to measure the percentage of time a Cassandra server was doing stop-the-world GC (Young Gen GC) and could not serve client requests.
Again, this metric doesn't tell you anything if you don't know how long each of the pauses are. If they are at the limit infinitesimally small then you are again only measuring the impact on throughput, not latency.
Certainly, GCs with long STW pauses do impact latency, but then you need to measure histograms of absolute pause times, not averages of ratios relative to application time. That's just a silly metric.
And neither does the article mention which JVM or GC they're using. Absent further information they might have gotten their 10x improvement relative to some especially poor choice of JVM and GC.
Earlier quoted context omitted.
I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.
> Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first. Huh? The AGPL is not a non-commercial-use-only license. If you have proprietary software that you would like to combine with AGPL code (i.e., not interact with as a service) and is available to the general public over the Internet, and you want keep your code proprietary, sure, you may not…
At the end of the day, it's not worth risking yourself (or your company) when the owners of the library claims a software license works a certain way and you disagree. Sure you might be right and you might even prevail in court, but the potential legal fees usually aren't worth the trouble.
I ran into this issue when I was selecting a library to generate PDFs for my internship over the summer: https://itextpdf.com/AGPL
As a Java scoffer trying to be fair-minded, I resisted the urge to joke that "it's was the GC, stupid" and assume that a big project like Cassandra had somehow worked around the GC latency problems. But, what? It turns out the article is really about replacing Java with C++.
They managed to generalize one application that meets their feature needs to be a front end to another existing application with fewer features but better performance as a back end. They're optimizing their hot path by decoupling it from the rest of the application and handing off to C++ code they didn't even have to write. Adding pluggable storage engines to Cassandra means that if they make the API smooth enough they can have engines in C, C++, Erlang, Go, Rust, ML, or whatever in the future without changing their front end. That's a big win even beyond this tail latency issue.
I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…