Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

Post reply on HN
Print view View original

41–50 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#41

Post by en4bz »

ghshephard wrote: post #32
en4bz wrote: post #4

Earlier quoted context omitted.

I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.

There are those who've deployed on Java with tight latency requirements: https://martinfowler.com/articles/lmax.html?t=1319912579 - Benchmarked at around 6 million transactions/second. The issue isn't so much Java the language, as it is being aware of the GC, and developing with it in mind.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#43

Post by estebank »

cmrdporcupine wrote: post #40

I remember using quite early versions of Cassandra back in an ad-tech startup I was at back in 2009 or 2010, spending unfortunate amounts of time fighting the JVM GC and trying to tune things so it behaved responsibly. It was a real problem then and I know a lot of work went into fixing GC behaviour. Then I stopped using Cassandra for work, but it's unfortunate this is still an issue? What I took out of that is that…

It sounds like you might be interested in TiKV.

https://github.com/pingcap/tikv

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#44

Post by mschuster91 »

en4bz wrote: post #41
ghshephard wrote: post #32

Earlier quoted context omitted.

There are those who've deployed on Java with tight latency requirements: https://martinfowler.com/articles/lmax.html?t=1319912579 - Benchmarked at around 6 million transactions/second. The issue isn't so much Java the language, as it is being aware of the GC, and developing with it in mind.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

> If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

Well, Java has the advantage of being platform (and to a certain degree, runtime) independent, plus a robust set of best practices and ecosystem when it comes to modules and library handling, which is pretty hard to get done right for C/C++ projects.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#45

Post by en4bz »

3uclid wrote: post #42

Unrelated: as a CS undergrad, I read this article and was immediately inspired. This is definitely the type of work I want to be doing when I graduate (infrastructure engineering). But my next thought was: where do I start?! Any advice?

CMU Database Group Lectures: https://www.youtube.com/channel/UCHnBsf2rH-K7pn09rb3qvkA

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#46

Post by amarkov »

en4bz wrote: post #41
ghshephard wrote: post #32

Earlier quoted context omitted.

There are those who've deployed on Java with tight latency requirements: https://martinfowler.com/articles/lmax.html?t=1319912579 - Benchmarked at around 6 million transactions/second. The issue isn't so much Java the language, as it is being aware of the GC, and developing with it in mind.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

Even in a database, most of the code isn't performance sensitive. Making your life a bit harder in the fast path so it's easier in the slow path is at least a tradeoff worth considering.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#47

Post by en4bz »

mschuster91 wrote: post #44
en4bz wrote: post #41

Earlier quoted context omitted.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

> If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++. Well, Java has the advantage of being platform (and to a certain degree, runtime) independent, plus a robust set of best practices and ecosystem when it comes to modules and library handling, which is pretty hard to get done right for C/C++ projects.

Node.js is also platform independent and has a package manager but I wouldn't use it for a High Performance / Low Latency application like a database.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#48

Post by geofft »

snovv_crash wrote: post #9
glommer wrote: post #6

Earlier quoted context omitted.

(ScyllaDB employee here) I don't believe one would need a commercial license just to test a product in any way? They are not making that part of any product at that point, so no concerns here.

They can't test on production servers. Fake data, non-userfacing servers, sure.

Instagram doesn't operate any user-facing Cassandra servers, though. They run user-facing web servers that talk to Cassandra internally.

I don't like the AGPL because it's unclear on this exact sort of thing, but it does seem to me like the obvious reading of "all users interacting with it remotely through a computer network" does not encompass the connection between Instagram end users and their internal Cassandra.

And, in any case, they released sources for the thing they came up with - which is all that the AGPL requires. If they're okay with doing that, they can definitely use the AGPL for production commercial software.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#49

Post by pas »

dikanggu wrote: post #38
pas wrote: post #27

Earlier quoted context omitted.

The JIRA tickets don't really shine with much hope :/ https://issues.apache.org/jira/browse/CASSANDRA-13474 [2 comments from 2017 Apr] https://issues.apache.org/jira/browse/CASSANDRA-13475 [~100 comments, but the last one is from 2017 Nov, by the InstaG engineer] And the Rocksandra fork is already ~3500 commits behind master, so upstreaming this will be interesting. Oh, and the Rocksandra fork is already kind of aban…

This is the rocksandra branch, https://github.com/Instagram/cassandra/tree/rocks_3.0 , we develop it on top of Cassandra 3.0. It's the code we are running on our production servers.

Thanks for the git push and the reply! A few minutes ago it was still pointing to the older commit.

And upstream Cassandra is already 11 minor releases away. Won't that become a problem with something as fundamental/low-level as pluggable storage engines?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#50

Post by cmrdporcupine »

estebank wrote: post #43
cmrdporcupine wrote: post #40

I remember using quite early versions of Cassandra back in an ad-tech startup I was at back in 2009 or 2010, spending unfortunate amounts of time fighting the JVM GC and trying to tune things so it behaved responsibly. It was a real problem then and I know a lot of work went into fixing GC behaviour. Then I stopped using Cassandra for work, but it's unfortunate this is still an issue? What I took out of that is that…

It sounds like you might be interested in TiKV. https://github.com/pingcap/tikv

Thanks.

Since coming to Google I don't get the opportunity to compare/evaluate/deploy tools like this anymore. Smarter people than me make choices like that :-)

Post reply on HN