Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

11–20 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#11
post #2

Weird, did they try to use https://www.scylladb.com/ ?

- has anyone run it FB scale? for how long?

- how many experienced scylladb devops are there globally that we can hire?

Those questions asked at BigTechCo before it adopts somebody elses tech.

FB already operates RocksDb and Cassandra so there's way less technical, career, financial risk for just hacking the two together with some aggressive refactoring.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#12
post #2

Weird, did they try to use https://www.scylladb.com/ ?

- has anyone run it FB scale? for how long? - how many experienced scylladb devops are there globally that we can hire? Those questions asked at BigTechCo before it adopts somebody elses tech. FB already operates RocksDb and Cassandra so there's way less technical, career, financial risk for just hacking the two together with some aggressive refactoring.

Does FB still use Cassandra? I thought they abandoned them ages ago and then databricks picked it up?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#14
post #2

Weird, did they try to use https://www.scylladb.com/ ?

Why throw away something proven to run at massive scale, that you understand and trust for something that's new, has never been run at that scale, and you have no experience running? If you have a team of software engineers, and the latency problem is a software problem, fix the software problem.

When you already know Cassandra, and you already know RocksDB, and you already have an engineering team, it makes far more sense to combine the two things you know how to use at scale than to try to use some new thing NOBODY has run at scale.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#16
post #4

Earlier quoted context omitted.

I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.

Doesn't AGPL allow commercial use?

Yes it does, but AGPL licenced software is super banned at all major companies because it is very viral. You have to make derivative software available under AGPL even if the end user accesses it only over a network.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#17
post #10

Earlier quoted context omitted.

They can't test on production servers. Fake data, non-userfacing servers, sure.

Even if that is a problem, we provide anyone that is interested with a 30-day evaluation license of Scylla Enterprise.

That puts you about 2/3rds of the way down the list of things to try. You're before all the products that have no evaluation license at all, but after all the open source options that developers can test at their leisure.

If I'm trying a products evaluation license, you can be sure I've tried literally every other option under the sun first, including investigating the possibility rolling my own if situationally appropriate. No form of development is slower than the kind where I have to wait for a company in another timezone to give me permission to use their software, so it's always last on my options list unless the company has frankly amazing reviews that pique my curiosity.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#18
Did you all find that there were changes to the Java heap/GC configuration that would make tuning this setup different? I imagine if most everything that "sticks" is moved off heap, the GC could be tuned more heavily for young gen throughput vs trying to balance it with long-lived objects.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#20
post #4
post #2

Weird, did they try to use https://www.scylladb.com/ ?

I was going to say the same thing. It seems pretty clear at this point that Java is not a good programming language to build a database on if you care about strong 99% latency guarantees. The engineers in the article came to this conclusion and so did the Scylla people years ago. Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first.

The server is AGPL. The client is Apache licensed. So I don't see a problem with using the AGPL version in commercial product.

Noone claims that a product using the MySql driver is a derivative work of the MySql server?

Edit: Of course, IANAL...

Post reply on HN