Open-sourcing a 10x reduction in Apache Cassandra tail latency • Hacker News Board

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#131

Post by majidazimi » Tue, Mar 06, 2018, 9:45 AM UTC

amarkov wrote: post #46

en4bz wrote: post #41

Earlier quoted context omitted.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

Even in a database, most of the code isn't performance sensitive. Making your life a bit harder in the fast path so it's easier in the slow path is at least a tradeoff worth considering.

Based on previous research [1], almost every part of a DB is in the hot path.

[1] http://15721.courses.cs.cmu.edu/spring2018/papers/02-inmemor...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#132

Post by ddorian43 » Tue, Mar 06, 2018, 9:48 AM UTC

3uclid wrote: post #97

ddorian43 wrote: post #89

Earlier quoted context omitted.

Still in school ? (don't understand different grad). See: GSOC Seastar Framework https://summerofcode.withgoogle.com/organizations/6190282903...

Yeah, still in school (3rd year). I have intern experience, but it seems like these type of positions are way too advanced for me at the moment. Just unsure how to progress...

1. see the other comment for CMU 2. learn c++ + data structures 3. try seastar (or any other lib?) and try to follow the tutorial 4. go to mailing list and ask for help when you're stuck 5. look into issue-queue for small tasks and grow little by little

Makes sense ? Another project you may look at is https://github.com/phaistos-networks/Trinity which is a library and should be simpler than a whole framework/db (trinity is like lucene/rocksdb compared to full-db cassandra/scylladb).

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#133

Post by ddorian43 » Tue, Mar 06, 2018, 9:59 AM UTC

ismail wrote: post #123

So question: Any thoughts on replacing HDFS + Yarn + Hive + HBASE with GulsterFS + Kubernetes + Cassandra ??

Hbase is sync+globally sorted, while cassandra is not, so probably not.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#134

Post by majidazimi » Tue, Mar 06, 2018, 10:01 AM UTC

tibbetts wrote: post #116

haglin wrote: post #61

"To reduce the GC impact from the storage engine, we considered different approaches and ultimately decided to develop a C++ storage engine to replace existing ones." I wonder how the numbers would have looked with the new low latency GC for Hotspot (ZGC). https://wiki.openjdk.java.net/display/zgc/Main Early results from SPECjbb2015 are impressive. https://youtu.be/tShc0dyFtgw?t=5m1s

Yes, also Azul Zing. Really anytime someone says they have a problem with GC and suggests spending a million dollars of engineer time building a new system, they should consider Zing first. It works and is a way more efficient way of spending money to fix GC latency problems.

Because GC related issues don't undergo from "problem" state to "solved" state. It is just a never ending stream of issues, that the team need to resolve, specially in a database realm in which metrics are hugely workload dependent.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#135

Post by nvarsj » Tue, Mar 06, 2018, 10:52 AM UTC

chii wrote: post #127

thefounder wrote: post #99

Earlier quoted context omitted.

Apparently these people enjoy GC/JVM languages more than C++.

GC languages like Java is much easier to write, and can be made performant when required.

Yes, this is the reason why Java is chosen. But I feel pretty strongly that databases are system engineering problems, and should be written using a proper systems language.

Something like Java makes implementation easier, but operation more difficult and costly.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#136

Post by nvarsj » Tue, Mar 06, 2018, 10:56 AM UTC

jjirsa wrote: post #105

nvarsj wrote: post #98

Earlier quoted context omitted.

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

To be fair, a lot of big companies know how to tune the JVM. A TON of HUGE companies write a LOT of java. What you consider a constant struggle, a lot of very large companies consider trivial.

I'm not sure it's trivial. Tuning the JVM is an entire cottage industry. JVM performance experts can make 1000+/day tuning the JVM and are in high demand. Companies spend huge amounts of engineering effort to keep the JVM running smoothly. I used to be involved in this side of things pretty heavily at a HFT firm, which almost exclusively used Java.

In my opinion it's a colossal waste of resources. Classic example of using the wrong tool for the job.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#137

Post by manigandham » Tue, Mar 06, 2018, 10:57 AM UTC

nvarsj wrote: post #98

_ivvf wrote: post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

Many of these open-source databases started as internal projects inside big companies, where Java/JVM allowed for more productivity and cross-platform deployment with more skill reuse of the team. Then they grew from there and now it's too late to rewrite the whole thing.

If you were starting a database-focused company from the beginning than choosing C++ is a better decision, which is exactly what ScyllaDB has done with their cassandra clone. Along with general algorithm and decision improvements, it'll provide 10-100x the same performance at lower latency on the same server.

We're also starting to see more projects written in Go now, which is still a managed runtime but usually better at handling these kinds of low-level systems.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#138

Post by manigandham » Tue, Mar 06, 2018, 11:00 AM UTC

jjirsa wrote: post #121

welder wrote: post #114

Earlier quoted context omitted.

> some new thing NOBODY has run at scale Outbrain uses ScyllaDB in production at scale across multiple data centers. Not sure if it's Instagram scale, but still enough to prove it's reliability and performance. https://www.outbrain.com/techblog/2016/08/scylladb-poc-not-s...

7 hosts in that poc, that is not "at scale"

Scylla can handle 10-100x the load of Cassandra on the same servers. Scale is more than just the number of hosts.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#139

Post by manigandham » Tue, Mar 06, 2018, 11:07 AM UTC

truth_seeker wrote: post #117

yazr wrote: post #83

Or just try and benchmark Azul VM with pause-less GCs ?! (I have used Azul in low-latency production environments. It has pros and cons but it certainly beats re-writing the storage layer... )

Curious to know the cons of using it, except being commercial.

> except being commercial

That's the biggest, especially for when it's a Facebook company. Otherwise it works well but can be pricey.

The JVM is getting a new fully concurrent collector though called Shenandoah: https://www.google.com/search?q=shenandoah+gc

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#140

Post by nvarsj » Tue, Mar 06, 2018, 11:07 AM UTC

manigandham wrote: post #137

nvarsj wrote: post #98

Earlier quoted context omitted.

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

Many of these open-source databases started as internal projects inside big companies, where Java/JVM allowed for more productivity and cross-platform deployment with more skill reuse of the team. Then they grew from there and now it's too late to rewrite the whole thing. If you were starting a database-focused company from the beginning than choosing C++ is a better decision, which is exactly what ScyllaDB has done…

Go is a much better choice for systems work. Largely because the GC has a low pause (sub ms) target. I'd still be hesitant to use it for very latency sensitive things, or memory intensive applications. Prometheus, for example, has struggled with golang's memory management (bad memory fragmentation, wasteful memory usage). But I think it's a great compromise if you don't want to deal with memory management.