Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

Post reply on HN
Print view View original

21–30 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#21

Post by wenc »

polskibus wrote: post #12
jakelarkin wrote: post #11

Earlier quoted context omitted.

- has anyone run it FB scale? for how long? - how many experienced scylladb devops are there globally that we can hire? Those questions asked at BigTechCo before it adopts somebody elses tech. FB already operates RocksDb and Cassandra so there's way less technical, career, financial risk for just hacking the two together with some aggressive refactoring.

Does FB still use Cassandra? I thought they abandoned them ages ago and then databricks picked it up?

> then databricks picked it up

I think it's DataStax. Databricks is the company behind Spark.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#23

Post by rbranson »

en4bz wrote: post #19

Has any tried running Casandra on Azul Zing[1]? The slowdown here is not surprisingly related to GC pauses which Azul has eliminated in Zing. [1] https://www.azul.com/products/zing/

The licensing cost of Zing generally makes this a bad trade-off. It's much cheaper to purchase more hardware. Zing is targeted at vertically scaling very large JVM heaps, where it's valuable to have massive amounts of data on a single, big machine.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#24

Post by jjirsa »

en4bz wrote: post #19

Has any tried running Casandra on Azul Zing[1]? The slowdown here is not surprisingly related to GC pauses which Azul has eliminated in Zing. [1] https://www.azul.com/products/zing/

Have friends who have used it, they report that it works reasonably well. Especially in p99.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#25

Post by jjirsa »

polskibus wrote: post #12
jakelarkin wrote: post #11

Earlier quoted context omitted.

- has anyone run it FB scale? for how long? - how many experienced scylladb devops are there globally that we can hire? Those questions asked at BigTechCo before it adopts somebody elses tech. FB already operates RocksDb and Cassandra so there's way less technical, career, financial risk for just hacking the two together with some aggressive refactoring.

Does FB still use Cassandra? I thought they abandoned them ages ago and then databricks picked it up?

The article is literally written by Instagram, which is FB.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#26

Post by openasocket »

I'm not an expert on these things, but it seems to me if you're implementing a database in Java you wouldn't want to keep your data on the JVM Heap, as this seems to indicate. My understanding is that in most applications (like servers) the average object lives for a very short period of time, and most GC implementations are built from that idea. But, in a database, especially an in-memory database, the majority of the objects are going to live for a very long time. That makes the mark phase of GC a lot more expensive, puts more pressure on the generations, etc.

Is my guess here correct, or are there things I'm missing or mistaken on?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#27

Post by pas »

jjirsa wrote: post #13

Nicely done! Looking forward to the pluggable storage engine.

The JIRA tickets don't really shine with much hope :/

https://issues.apache.org/jira/browse/CASSANDRA-13474 [2 comments from 2017 Apr] https://issues.apache.org/jira/browse/CASSANDRA-13475 [~100 comments, but the last one is from 2017 Nov, by the InstaG engineer]

And the Rocksandra fork is already ~3500 commits behind master, so upstreaming this will be interesting.

Oh, and the Rocksandra fork is already kind of abandoned - no commits since 2017 Dec. (which probably means this is not actually the code that runs under Instagram.)

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#28

Post by dikanggu »

rbranson wrote: post #18

Did you all find that there were changes to the Java heap/GC configuration that would make tuning this setup different? I imagine if most everything that "sticks" is moved off heap, the GC could be tuned more heavily for young gen throughput vs trying to balance it with long-lived objects.

Yeah, for Rocksandra, we are able to use much smaller heap size, and most of the objects are recycled during the young gen GC.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#29

Post by spockz »

en4bz wrote: post #19

Has any tried running Casandra on Azul Zing[1]? The slowdown here is not surprisingly related to GC pauses which Azul has eliminated in Zing. [1] https://www.azul.com/products/zing/

Actually, it appears that is one of the premises[1] they sell Zing on.

[1]: https://www.azul.com/solutions/cassandra/

Post reply on HN