Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

Post reply on HN
Print view View original

51–60 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#51

Post by jjirsa »

kiril-me wrote: post #30
jjirsa wrote: post #13

Nicely done! Looking forward to the pluggable storage engine.

It would be great. But I don't think it could happen. The pluggable storage engine would greatly increase the cognitive complexity of the code.

Of course it could happen. Pluggable engine has a lot of positives - not only does it enable features like this, it also helps modularize the codebase making it more testable, and there are other people who will develop storage engines for their own use case over time (look at the evolution of - for example - MySQL storage backends for examples of this).

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#53

Post by geofft »

en4bz wrote: post #4
Thaxll wrote: post #2

Weird, did they try to use https://www.scylladb.com/ ?

I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.

> Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.

Huh? The AGPL is not a non-commercial-use-only license.

If you have proprietary software that you would like to combine with AGPL code (i.e., not interact with as a service) and is available to the general public over the Internet, and you want keep your code proprietary, sure, you may not want to use the AGPL. But you could say the same thing about proprietary software you want to combine with GPL code and sell to the general public.

If you're either using the software through it's existing defined public interfaces, or you're okay releasing anything you modify or link into the software, the AGPL (and the GPL) are fine. Lots of people distribute proprietary products that include GPL code, like Chromebooks, Android phones, routers, GitHub Enterprise, etc. We figured out years ago that the Linux kernel is not just a non-commercial product. Why are we having the same misconceptions about the AGPL?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#54

Post by jjirsa »

3uclid wrote: post #42

Unrelated: as a CS undergrad, I read this article and was immediately inspired. This is definitely the type of work I want to be doing when I graduate (infrastructure engineering). But my next thought was: where do I start?! Any advice?

Happy to help you get started working on Cassandra. http://cassandra.apache.org/doc/latest/development/patches.h... Has some basic entry pointers. There’s also a dev mailing list that’s reasonable active.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#55

Post by pas »

jjirsa wrote: post #37
pas wrote: post #27

Earlier quoted context omitted.

The JIRA tickets don't really shine with much hope :/ https://issues.apache.org/jira/browse/CASSANDRA-13474 [2 comments from 2017 Apr] https://issues.apache.org/jira/browse/CASSANDRA-13475 [~100 comments, but the last one is from 2017 Nov, by the InstaG engineer] And the Rocksandra fork is already ~3500 commits behind master, so upstreaming this will be interesting. Oh, and the Rocksandra fork is already kind of aban…

I'm a committer, I'm familiar with the JIRA ticket.

Could you share your thoughts on how likely and how soon will the RocksDB engine be available as part of normal Cassandra? Also, how big is the gap between 3.0.x and 3.x? Any improvements between 3.0.x and 3.x regarding tail latency/performance? Thanks!

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#56

Post by jjirsa »

pas wrote: post #55
jjirsa wrote: post #37

Earlier quoted context omitted.

I'm a committer, I'm familiar with the JIRA ticket.

Could you share your thoughts on how likely and how soon will the RocksDB engine be available as part of normal Cassandra? Also, how big is the gap between 3.0.x and 3.x? Any improvements between 3.0.x and 3.x regarding tail latency/performance? Thanks!

I think it's likely. It's decidedly nontrivial, and the hardest part will be the (very slow) design phase where we actually make sure the interfaces are defined properly, but I think there are enough interested people to make sure it happens.

There are some meaningful changes between 3.0 and 3.11 (notably a compressed chunk cache for storing some intermediate data blocks and a significant change to the way the column index is deserialized) that do help tail latencies, and there's certainly quite a bit more low hanging fruit, but the biggest contributor to p99 latencies is the GC collections, and the read path still contributes the most JVM garbage, so this is still probably a meaningful improvement over 3.11.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#57

Post by en4bz »

geofft wrote: post #53
en4bz wrote: post #4

Earlier quoted context omitted.

I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.

> Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first. Huh? The AGPL is not a non-commercial-use-only license. If you have proprietary software that you would like to combine with AGPL code (i.e., not interact with as a service) and is available to the general public over the Internet, and you want keep your code proprietary, sure, you may not…

I think you answered the question yourself [1].

The wording is ambiguous and as far as I know there have been no court cases yet that have yet to define what constitutes a connection between the end user and whether transitive connections count. If it's ambiguous to a software developer then corporate lawyers are definitely going to say no.

[1] https://news.ycombinator.com/item?id=16523858

EDIT: I realize that AGPL is valid for commercial use but since its terms are so onerous, especially once the lawyers get involved, it effectively makes the AGPL unusable in a larger corporation.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#58

Post by openasocket »

wonnage wrote: post #36
openasocket wrote: post #26

I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…

The purpose of separating into young and old generation is that it's easier to find dead objects in the young generation (as you said, average object lives for a short period of time). You only have to scan this subset for a minor GC. It doesn't really matter how many long-lived objects you have as long as you can avoid needing to do a major GC.

Don't you still need to scan the old generation during minor GC, in case a field in one of the older objects was modified to point to an object in the young generation? Or are there optimizations you can use to quickly and efficiently find references from the older generation to the younger?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#59

Post by en4bz »

geofft wrote: post #48
snovv_crash wrote: post #9

Earlier quoted context omitted.

They can't test on production servers. Fake data, non-userfacing servers, sure.

Instagram doesn't operate any user-facing Cassandra servers, though. They run user-facing web servers that talk to Cassandra internally. I don't like the AGPL because it's unclear on this exact sort of thing, but it does seem to me like the obvious reading of "all users interacting with it remotely through a computer network" does not encompass the connection between Instagram end users and their internal Cassandra.…

Counldn't sticking a proxy in front of any AGPL software defeat its purpose then? If you don't consider transitive connections it seems pointless to me.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#60

Post by adrianratnapala »

As a Java scoffer trying to be fair-minded, I resisted the urge to joke that "it's was the GC, stupid" and assume that a big project like Cassandra had somehow worked around the GC latency problems.

But, what? It turns out the article is really about replacing Java with C++.

Post reply on HN