Open-sourcing a 10x reduction in Apache Cassandra tail latency

Post by manigandham » Tue, Mar 06, 2018, 11:10 AM UTC

nvarsj wrote: post #140

manigandham wrote: post #137

Earlier quoted context omitted.

Many of these open-source databases started as internal projects inside big companies, where Java/JVM allowed for more productivity and cross-platform deployment with more skill reuse of the team. Then they grew from there and now it's too late to rewrite the whole thing. If you were starting a database-focused company from the beginning than choosing C++ is a better decision, which is exactly what ScyllaDB has done…

Go is a much better choice for systems work. Largely because the GC has a low pause (sub ms) target. I'd still be hesitant to use it for very latency sensitive things, or memory intensive applications. Prometheus, for example, has struggled with golang's memory management (bad memory fragmentation, wasteful memory usage). But I think it's a great compromise if you don't want to deal with memory management.

Post by eptcyka » Tue, Mar 06, 2018, 12:16 PM UTC

b4lancesh33t wrote: post #101

_ivvf wrote: post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

> Given that stop-the-world pauses are never evenly distributed throughout time That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latenc…

Post by cnlwsu » Tue, Mar 06, 2018, 1:29 PM UTC

manigandham wrote: post #138

jjirsa wrote: post #121

Earlier quoted context omitted.

7 hosts in that poc, that is not "at scale"

Scylla can handle 10-100x the load of Cassandra on the same servers. Scale is more than just the number of hosts.

Post by jakewins » Tue, Mar 06, 2018, 1:46 PM UTC

ndesaulniers wrote: post #112

jakewins wrote: post #39

Earlier quoted context omitted.

This is correct; the standard approach here is to use regular c-style memory management for the data the system is managing, and the JVM heap only for the database "infrastructure". This hybrid approach gives the benefit of a managed runtime and safety of GC for most of your code, but allows the performance of raw pointers/malloc for key code paths. Some examples of this pattern on the JVM: - The Neo4j Page Cache, Mu…

> but allows the performance of ... malloc for key code paths. Everything is relative, I guess.

Post by manigandham » Tue, Mar 06, 2018, 4:18 PM UTC

cnlwsu wrote: post #145

manigandham wrote: post #138

Earlier quoted context omitted.

Scylla can handle 10-100x the load of Cassandra on the same servers. Scale is more than just the number of hosts.

Data density is a thing. If u putting 10tb on a c* host, switching to Scylla doesn’t fix the issues that putting 1pb of data on a host would involve (ie backing that up). Throughput of 100mb of data done in marketing benchmarks are rarely relevant.