Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

Post reply on HN
Print view View original

111–120 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#111

Post by discoursism »

foolfoolz wrote: post #78
the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

classic hacker news comment. this thing you built and open sourced, has gotten you real measurable results? allow me to list the many ways you’re probably wrong and doing it incorrectly

Measurable results are all well and good, but it can be helpful to know how the baseline was established. Measurable results aren't "portable" without a well-established baseline.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#112

Post by ndesaulniers »

jakewins wrote: post #39
openasocket wrote: post #26

I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…

This is correct; the standard approach here is to use regular c-style memory management for the data the system is managing, and the JVM heap only for the database "infrastructure". This hybrid approach gives the benefit of a managed runtime and safety of GC for most of your code, but allows the performance of raw pointers/malloc for key code paths. Some examples of this pattern on the JVM: - The Neo4j Page Cache, Mu…

> but allows the performance of ... malloc for key code paths.

Everything is relative, I guess.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#114

Post by welder »

jjirsa wrote: post #14
Thaxll wrote: post #2

Weird, did they try to use https://www.scylladb.com/ ?

Why throw away something proven to run at massive scale, that you understand and trust for something that's new, has never been run at that scale, and you have no experience running? If you have a team of software engineers, and the latency problem is a software problem, fix the software problem. When you already know Cassandra, and you already know RocksDB, and you already have an engineering team, it makes far more…

> some new thing NOBODY has run at scale

Outbrain uses ScyllaDB in production at scale across multiple data centers. Not sure if it's Instagram scale, but still enough to prove it's reliability and performance.

https://www.outbrain.com/techblog/2016/08/scylladb-poc-not-s...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#115

Post by tibbetts »

archi42 wrote: post #90
zenonu wrote: post #82

Earlier quoted context omitted.

Exactly. Immediately from the premise of the paper, I was looking forward to a discussion on how they tried various strategies to tune their JVM/GC parameters and found nothing. The "well I guess we gotta replace this with a C++ solution" sentiment smacks of poor software engineering practice, despite the results.

Good memory management is important for a high performance, low latency DBMS. I don't see how getting rid of GC is bad engineering practice. If a tool is not well suited for a problem, use another tool. Could you please explain your opinion? //Edit: removed 2nd part, which wasn't really that important anyway...

That might be an argument for not using Cassandra. It’s a pretty big leap to reimpmenting half of Cassandra in c++.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#116

Post by tibbetts »

haglin wrote: post #61

"To reduce the GC impact from the storage engine, we considered different approaches and ultimately decided to develop a C++ storage engine to replace existing ones." I wonder how the numbers would have looked with the new low latency GC for Hotspot (ZGC). https://wiki.openjdk.java.net/display/zgc/Main Early results from SPECjbb2015 are impressive. https://youtu.be/tShc0dyFtgw?t=5m1s

Yes, also Azul Zing. Really anytime someone says they have a problem with GC and suggests spending a million dollars of engineer time building a new system, they should consider Zing first. It works and is a way more efficient way of spending money to fix GC latency problems.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#117

Post by truth_seeker »

yazr wrote: post #83

Or just try and benchmark Azul VM with pause-less GCs ?! (I have used Azul in low-latency production environments. It has pros and cons but it certainly beats re-writing the storage layer... )

Curious to know the cons of using it, except being commercial.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#118

Post by tibbetts »

itronitron wrote: post #102
en4bz wrote: post #41

Earlier quoted context omitted.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

developing in Java with awareness of the GC doesn't mean tracking allocation and deallocation of memory, it means developers should avoid allocating lots of new Objects when possible. In practice this means creating view, cursor, or offset type Objects that map to arrays of more primitive data types.

Exactly. Developing GC aware code is still easier and safer than c++ memory management. Particularly because when you screw up in c++ you get crashes or data loss, and when you screw up in Java you mostly just get GC pauses.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#119

Post by truth_seeker »

jjirsa wrote: post #24
en4bz wrote: post #19

Has any tried running Casandra on Azul Zing[1]? The slowdown here is not surprisingly related to GC pauses which Azul has eliminated in Zing. [1] https://www.azul.com/products/zing/

Have friends who have used it, they report that it works reasonably well. Especially in p99.

By what factor/magnitude p99 was improved ? Any idea ?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#120

Post by mlla »

nvarsj wrote: post #98
_ivvf wrote: post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

Good Q. You might like to check ScyllaDB written in C++, which is supposed to have considerably better performance than Cassandra (also low tail-latency) and a level of compatibility with it: https://www.scylladb.com/
Post reply on HN