Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

101–110 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#101
post #95
post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

> Given that stop-the-world pauses are never evenly distributed throughout time

That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latency99 due to GC. That would entail 5 of 25 pauses happening during a 20ms period in a 1s window.

(This idea is not purely hypothetical. For example, Go's GC has very low STW periods.)

> It's pretty much accepted everywhere

Eh. Apparently everyone thinks C is the best language for cryptography and other secure but not particularly perf sensitive code. Go figure. Sometimes the wisdom of the masses is not wisdom. Best not to appeal to it during argumentation.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#102
post #41

Earlier quoted context omitted.

There are those who've deployed on Java with tight latency requirements: https://martinfowler.com/articles/lmax.html?t=1319912579 - Benchmarked at around 6 million transactions/second. The issue isn't so much Java the language, as it is being aware of the GC, and developing with it in mind.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

developing in Java with awareness of the GC doesn't mean tracking allocation and deallocation of memory, it means developers should avoid allocating lots of new Objects when possible. In practice this means creating view, cursor, or offset type Objects that map to arrays of more primitive data types.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#103

Earlier quoted context omitted.

This is correct; the standard approach here is to use regular c-style memory management for the data the system is managing, and the JVM heap only for the database "infrastructure". This hybrid approach gives the benefit of a managed runtime and safety of GC for most of your code, but allows the performance of raw pointers/malloc for key code paths. Some examples of this pattern on the JVM: - The Neo4j Page Cache, Mu…

If so any JVM based datastore could probably benefit. I wonder how long before we see a similar result from ElasticSearch. (Only other huge JVM based store I can think of).

People with an interest may want to check out Azul's Zing JVM and the following blog post testing it out with Lucene... http://blog.mikemccandless.com/2012/07/lucene-index-in-ram-w...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#104
post #61

"To reduce the GC impact from the storage engine, we considered different approaches and ultimately decided to develop a C++ storage engine to replace existing ones." I wonder how the numbers would have looked with the new low latency GC for Hotspot (ZGC). https://wiki.openjdk.java.net/display/zgc/Main Early results from SPECjbb2015 are impressive. https://youtu.be/tShc0dyFtgw?t=5m1s

yes, a comparison across multiple JVMs would be nice

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#105
post #98
post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

To be fair, a lot of big companies know how to tune the JVM. A TON of HUGE companies write a LOT of java. What you consider a constant struggle, a lot of very large companies consider trivial.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#107

Earlier quoted context omitted.

The exact interpretation varies from company to company. Some companies take the strict stance of "if you use this library in any way in your application, you must open source your entire application." I've found that some libraries explicitly state that requirement within their FAQs for their community/free edition as opposed to their commercially (and paid) licensed equivalent. At the end of the day, it's not worth…

FWIW, I’m pretty sure Facebook also uses mongodb heavily (or at least through its acquisition of parse) I don’t know if they have a commercial license or not, but they aren’t strangers to the AGPL

MongoDB also makes their interpretation of the AGPL pretty well known - unfortunately it’s at odds with established interpretation of the normal GPL.

Requiring GPL/AGPL software as a dependency even if you don’t link to it, but instead talk to it over the network does not mean you haven’t developed a derivative work in terms of the letter and spirit of the license. This is in-part why I steer 100% clear of MongoDB, there’s nothing stopping them from changing their view on the license and deciding to pursue legal action against people who use it in non-AGPL compatible manners down the road.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#108
post #88
post #84

Earlier quoted context omitted.

That's specifically what the AGPL does (as opposed to the GPL.) The copyleft "infection" is deliberately transmitted via network clients, not just static linking. So you actually can't release a permissively licensed client for an AGPL server. I mean, they did, clearly, but the AGPL itself would seem to make that inconsistent. But then none of this has ever been litigated and both the AGPL and GPL themselves are very…

As I understand it, with the GPL, you must offer source code under the GPL to everyone you distribute the software to. With the AGPL, the same goes for those that use the software over the network. So you must offer the source of the database everyone who connects to the database over the network, under the AGPL. But if you deliver a web app, not a database-as-a-service, your users don't connect to the database. And…

Requiring GPL’d software to function means your product is a derivative work, full stop as far as the spirit and letter of the license is concerned. Using it over a network instead of linking against it doesn’t change this, if you depend on MySQL or any of the forks and distribute your software it must be made available under a GPL-compatible license. The requirements of the AGPL become clear in this regard as well; network use is distribution with the AGPL - incorporating AGPL’d software into your application means you must consider your entire application as licensed under the AGPL or compatible license.

MongoDB muddied the waters here by deciding to interpret the AGPL differently, but I wouldn’t risk your business on it.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#109

I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…

For a long time, the guidance was to install jemalloc and then use off-heap objects. I can’t recall what it was, but that broke in the 3.0.x series and is unlikely to return. The feature stream (3.1.x) allegedly can use jemalloc again, but we’ve been slow to adopt it so I can’t provide proof.
Post reply on HN