Can we add lz4 to the blend to reduce disk IO?
Open-sourcing a 10x reduction in Apache Cassandra tail latency
161–170 of 171 posts
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#162> We also observed that the GC stalls on that cluster dropped from 2.5% to 0.3%, which was a 10X reduction!
Umm .. shouldn't the stalls go to 0, because now you have moved to C++ ? Or is this the time it takes for the manual garbage collection to occur ?
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#163Earlier quoted context omitted.
I bet that didn't take N engineers 12 months to build out, either
Cassandra uses G1GC by default. If it was as simple as tweaking a few GC settings to get 10x improvement pretty sure Datastax would've done it by now.
Not sure you understand that there are different versions of Cassandra and I never mentioned that we used Datastax version. Using G1GC with default settings does not give you anything btw.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#164Or just try and benchmark Azul VM with pause-less GCs ?! (I have used Azul in low-latency production environments. It has pros and cons but it certainly beats re-writing the storage layer... )
[deleted]
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#165Or just try and benchmark Azul VM with pause-less GCs ?! (I have used Azul in low-latency production environments. It has pros and cons but it certainly beats re-writing the storage layer... )
Curious to know the cons of using it, except being commercial.
Needs a stronger machine to be effective (more cores & more memory)
Minor configuration issues (we had a very complex environment, custom kernel, weird network stuff, JNIs)
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#166Unrelated: as a CS undergrad, I read this article and was immediately inspired. This is definitely the type of work I want to be doing when I graduate (infrastructure engineering). But my next thought was: where do I start?! Any advice?
I'd say no matter what kind of job you get, you can put 10% of your time into similar problems. Even simple CRUD apps can have interesting problems like this. In my experience every project has instances of engineers shooting themselves in the foot, or unforeseen problems cropping up. If you have a bit of self-motivation you can dig into them and learn a lot and improve things. I do this and find it very satisfying.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#167Earlier quoted context omitted.
To be fair, a lot of big companies know how to tune the JVM. A TON of HUGE companies write a LOT of java. What you consider a constant struggle, a lot of very large companies consider trivial.
I'm not sure it's trivial. Tuning the JVM is an entire cottage industry. JVM performance experts can make 1000+/day tuning the JVM and are in high demand. Companies spend huge amounts of engineering effort to keep the JVM running smoothly. I used to be involved in this side of things pretty heavily at a HFT firm, which almost exclusively used Java. In my opinion it's a colossal waste of resources. Classic example of…
And it's still cheaper to hire a guy with a skill like that for a few weeks, or even keep him permanently - and keep a larger development team of cheaper C# or Java devs, than it is to replace them all with higher-payed C/C++ devs which would probably take longer to get the same functionality up & running.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#168I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of t…
For a long time, the guidance was to install jemalloc and then use off-heap objects. I can’t recall what it was, but that broke in the 3.0.x series and is unlikely to return. The feature stream (3.1.x) allegedly can use jemalloc again, but we’ve been slow to adopt it so I can’t provide proof.
The good news is, at least for my workloads, garbage collection is significantly better in 3.0 than 2.1 even without off-heap objects. Not sure if it's generating less or just generating it in a way that's easier to collect cheaply, but I saw pause times and total collection work drop significantly with the same settings (G1 collector)
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#169"To reduce the GC impact from the storage engine, we considered different approaches and ultimately decided to develop a C++ storage engine to replace existing ones." I wonder how the numbers would have looked with the new low latency GC for Hotspot (ZGC). https://wiki.openjdk.java.net/display/zgc/Main Early results from SPECjbb2015 are impressive. https://youtu.be/tShc0dyFtgw?t=5m1s
Yes, also Azul Zing. Really anytime someone says they have a problem with GC and suggests spending a million dollars of engineer time building a new system, they should consider Zing first. It works and is a way more efficient way of spending money to fix GC latency problems.
For a small to medium sized shop, sure. For someplace with thousands or tens of thousands of nodes, the new system ends up cheaper in the long run.
Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency
#170Earlier quoted context omitted.
- has anyone run it FB scale? for how long? - how many experienced scylladb devops are there globally that we can hire? Those questions asked at BigTechCo before it adopts somebody elses tech. FB already operates RocksDb and Cassandra so there's way less technical, career, financial risk for just hacking the two together with some aggressive refactoring.
Does FB still use Cassandra? I thought they abandoned them ages ago and then databricks picked it up?
FB abandoned Cassandra (which was really only used for message inbox indexing) when they redid how messages work years ago, but the re-adopted a large C* infra when they bought Instagram.