Open-sourcing a 10x reduction in Apache Cassandra tail latency • Hacker News Board

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#81

Post by dtparr » Mon, Mar 05, 2018, 10:00 PM UTC

teacpde wrote: post #80

the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

> If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. I try to understand the meaning. Is it saying the latency caused be GC is applied to all requests, not just the ones that observe 99th percentile latency?

It's saying that whether it affects latency or just throughput depends on how those pauses are distributed in absolute terms, not just the ratio. There's a big difference in 99th percentile latency between a 1ms pause every 400ms and a 10 second pause every 67 minutes, but they both work out to 2.5% by the ratio metric.

So yes, at the `infinitesimally small` end, time would be 'stolen' evenly from all request threads and would not be a contributing factor to the 99th percentile.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#82

Post by zenonu » Mon, Mar 05, 2018, 10:02 PM UTC

viraptor wrote: post #79

fdeliege wrote: post #71

Earlier quoted context omitted.

Both code and benchmark are open sourced. We'd love to hear how it performs for you.

This is a valid criticism of the methodology / explanation. It's not about the results. You can agree with the positive results (and they're great! - you've done awesome work and clearly show an improvement) and still say the explanation how/why they were achieved is not great.

Exactly. Immediately from the premise of the paper, I was looking forward to a discussion on how they tried various strategies to tune their JVM/GC parameters and found nothing. The "well I guess we gotta replace this with a C++ solution" sentiment smacks of poor software engineering practice, despite the results.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#83

Post by yazr » Mon, Mar 05, 2018, 10:02 PM UTC

Or just try and benchmark Azul VM with pause-less GCs ?!

(I have used Azul in low-latency production environments. It has pros and cons but it certainly beats re-writing the storage layer... )

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#84

Post by lukev » Mon, Mar 05, 2018, 10:02 PM UTC

gnud wrote: post #20

en4bz wrote: post #4

Earlier quoted context omitted.

I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.

The server is AGPL. The client is Apache licensed. So I don't see a problem with using the AGPL version in commercial product. Noone claims that a product using the MySql driver is a derivative work of the MySql server? Edit: Of course, IANAL...

That's specifically what the AGPL does (as opposed to the GPL.) The copyleft "infection" is deliberately transmitted via network clients, not just static linking.

So you actually can't release a permissively licensed client for an AGPL server. I mean, they did, clearly, but the AGPL itself would seem to make that inconsistent.

But then none of this has ever been litigated and both the AGPL and GPL themselves are very confusingly worded so shrug.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#85

Post by ddorian43 » Mon, Mar 05, 2018, 10:04 PM UTC

ddorian43 wrote: post #74

tkahnoski wrote: post #64

Earlier quoted context omitted.

If so any JVM based datastore could probably benefit. I wonder how long before we see a similar result from ElasticSearch. (Only other huge JVM based store I can think of).

Hbase is going offheap as much as possible. Voltdb uses java for management and c++ for low-level. They will write c++ in java eventually. Depending on how much performance you REALLY need. The same for elasticseach, if you want performance you need to do the same thing scylladb did to cassandra (per-core-sharding, skip filesystem across cores etc)

In elasticsearcch terms, vespa.ai, which claims better performance/scalability/maintanability uses c++ for lucene layer and java for the solr/elasticsearch layer.

There are blog posts speeding lucene by 2x+ by changing some stuff to c/c++. There are libraries (trinity) claiming 2x+ performance .

There is google-engineer saying "bigtable is 3x faster than hbase" that I've read.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#86

Post by the8472 » Mon, Mar 05, 2018, 10:05 PM UTC

teacpde wrote: post #80

the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

> If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. I try to understand the meaning. Is it saying the latency caused be GC is applied to all requests, not just the ones that observe 99th percentile latency?

No, that would be an incremental GC working in very small time slices.

A concurrent GC spends CPU cycles on different cores to do its work, which means it will not cause latency outliers in the threads processing the requests. They are still CPU cycles you don't have to serve other requests, hence they still affect throughput.

That is a simplified explanation of course, there are a lot of caveats.

In my original post I was mostly speaking about the measurement though, since they are measuring throughput when they are concerned about latency, those are somewhat related but depending on circumstances only weakly so.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#87

Post by viraptor » Mon, Mar 05, 2018, 10:05 PM UTC

zenonu wrote: post #82

viraptor wrote: post #79

Earlier quoted context omitted.

This is a valid criticism of the methodology / explanation. It's not about the results. You can agree with the positive results (and they're great! - you've done awesome work and clearly show an improvement) and still say the explanation how/why they were achieved is not great.

Exactly. Immediately from the premise of the paper, I was looking forward to a discussion on how they tried various strategies to tune their JVM/GC parameters and found nothing. The "well I guess we gotta replace this with a C++ solution" sentiment smacks of poor software engineering practice, despite the results.

That you didn't find it, doesn't mean they didn't do any. Maybe it wasn't worth mentioning. Maybe they wanted to keep the post short. Some team successfully did a storage engine transplant and you're saying they're doing poor engineering?

That's definitely not what I was suggesting in my comment.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#88

Post by gnud » Mon, Mar 05, 2018, 10:20 PM UTC

lukev wrote: post #84

gnud wrote: post #20

Earlier quoted context omitted.

The server is AGPL. The client is Apache licensed. So I don't see a problem with using the AGPL version in commercial product. Noone claims that a product using the MySql driver is a derivative work of the MySql server? Edit: Of course, IANAL...

That's specifically what the AGPL does (as opposed to the GPL.) The copyleft "infection" is deliberately transmitted via network clients, not just static linking. So you actually can't release a permissively licensed client for an AGPL server. I mean, they did, clearly, but the AGPL itself would seem to make that inconsistent. But then none of this has ever been litigated and both the AGPL and GPL themselves are very…

As I understand it, with the GPL, you must offer source code under the GPL to everyone you distribute the software to. With the AGPL, the same goes for those that use the software over the network.

So you must offer the source of the database everyone who connects to the database over the network, under the AGPL. But if you deliver a web app, not a database-as-a-service, your users don't connect to the database. And since this database uses the Cassandra protocol, I'd say your web app isn't a derived work of the database in any way.

Of course, that last part is the sticky bit. But if applications using database servers via a well defined protocol are judged to be derived works, we might have other problems - hence the reference to MySql in my first post.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#89

Post by ddorian43 » Mon, Mar 05, 2018, 10:25 PM UTC

3uclid wrote: post #42

Unrelated: as a CS undergrad, I read this article and was immediately inspired. This is definitely the type of work I want to be doing when I graduate (infrastructure engineering). But my next thought was: where do I start?! Any advice?

Still in school ? (don't understand different grad). See: GSOC Seastar Framework https://summerofcode.withgoogle.com/organizations/6190282903...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#90

Post by archi42 » Mon, Mar 05, 2018, 10:32 PM UTC

zenonu wrote: post #82

viraptor wrote: post #79

Earlier quoted context omitted.

This is a valid criticism of the methodology / explanation. It's not about the results. You can agree with the positive results (and they're great! - you've done awesome work and clearly show an improvement) and still say the explanation how/why they were achieved is not great.

Exactly. Immediately from the premise of the paper, I was looking forward to a discussion on how they tried various strategies to tune their JVM/GC parameters and found nothing. The "well I guess we gotta replace this with a C++ solution" sentiment smacks of poor software engineering practice, despite the results.

Good memory management is important for a high performance, low latency DBMS. I don't see how getting rid of GC is bad engineering practice. If a tool is not well suited for a problem, use another tool. Could you please explain your opinion?

//Edit: removed 2nd part, which wasn't really that important anyway...