Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

Post reply on HN
Print view View original

71–80 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#71

Post by fdeliege »

the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

Both code and benchmark are open sourced. We'd love to hear how it performs for you.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#72

Post by the8472 »

StreamBright wrote: post #70
openasocket wrote: post #26

I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…

If you want to keep data off heap you need to use sun.misc.Unsafe and allocate / free by yourself. I guess it is called unsafe for a reason. With G1GC you can do magical things to reduce the GC overhead which I always recommend as the first step before trying off heap.

You don't even have to go through Unsafe. ByteBuffer.allocateDirect() gives you a chunk of off-heap memory.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#73

Post by kodablah »

StreamBright wrote: post #70
openasocket wrote: post #26

I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…

If you want to keep data off heap you need to use sun.misc.Unsafe and allocate / free by yourself. I guess it is called unsafe for a reason. With G1GC you can do magical things to reduce the GC overhead which I always recommend as the first step before trying off heap.

ByteBuffer.allocateDirect is another off-JVM-heap solution that's not marked unsafe.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#74

Post by ddorian43 »

tkahnoski wrote: post #64
jakewins wrote: post #39

Earlier quoted context omitted.

This is correct; the standard approach here is to use regular c-style memory management for the data the system is managing, and the JVM heap only for the database "infrastructure". This hybrid approach gives the benefit of a managed runtime and safety of GC for most of your code, but allows the performance of raw pointers/malloc for key code paths. Some examples of this pattern on the JVM: - The Neo4j Page Cache, Mu…

If so any JVM based datastore could probably benefit. I wonder how long before we see a similar result from ElasticSearch. (Only other huge JVM based store I can think of).

Hbase is going offheap as much as possible. Voltdb uses java for management and c++ for low-level.

They will write c++ in java eventually. Depending on how much performance you REALLY need.

The same for elasticseach, if you want performance you need to do the same thing scylladb did to cassandra (per-core-sharding, skip filesystem across cores etc)

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#75

Post by tschellenbach »

For Stream's feed tech we also moved from Cassandra to an in-house solution on top of RocksDB. It's been a massive performance and maintenance improvement. This StackShare explains how Stream's stack works. It's based on Go, RocksDB and Raft: https://stackshare.io/stream/stream-and-go-news-feeds-for-ov...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#76

Post by russellspitzer »

dikanggu wrote: post #34

We do want to contribute our work back to the Cassandra upstream, instead of keeping it as a fork. So that more users from C* community can benefit from the improvements. The pluggable storage engine is an ambitious project ( https://issues.apache.org/jira/browse/CASSANDRA-13474 ). Any help will be appreciated!

Saw you talking about this on the Distributed Data Show

https://academy.datastax.com/content/distributed-data-show-e...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#77

Post by jjirsa »

openasocket wrote: post #26

I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…

Cassandra does a bunch of stuff off-heap - we keep things like Bloom Filters, our compression offsets (to seek into compressed data files), and even some of the memtable (the in-memory buffer before flushing) in direct memory, primary for the reasons you describe.

We still have "other" things on-heap. The biggest contributor to GC pain tends to be the number of objects allocated on the read path, so this patch works around that by pushing much of that logic to rocksdb.

There are certainly other things you can do in the code itself that would also help - one of the biggest contributors to garbage is the column index. CASSANDRA-9754 fixes much of that (jira is inactive, but the development work on it is ongoing).

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#78

Post by foolfoolz »

the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

classic hacker news comment.

this thing you built and open sourced, has gotten you real measurable results? allow me to list the many ways you’re probably wrong and doing it incorrectly

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#79

Post by viraptor »

fdeliege wrote: post #71
the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

Both code and benchmark are open sourced. We'd love to hear how it performs for you.

This is a valid criticism of the methodology / explanation. It's not about the results. You can agree with the positive results (and they're great! - you've done awesome work and clearly show an improvement) and still say the explanation how/why they were achieved is not great.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#80

Post by teacpde »

the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

> If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency.

I try to understand the meaning. Is it saying the latency caused be GC is applied to all requests, not just the ones that observe 99th percentile latency?

Post reply on HN