Scylla – Real-Time Big Data Database • Hacker News Board

Re: Scylla – Real-Time Big Data Database

#21

Post by manigandham » Tue, Aug 24, 2021, 11:17 PM UTC

kasey_junk wrote: post #8

criticaltinker wrote: post #6

The benchmarks against DynamoDB, Bigtable, & CockroachDB [1] appear quite impressive - anyone have real world experience that can attest to these claims of improved performance and reduced cost? > Scylla vs DynamoDB – Database Benchmark > 20x better throughput in the hot-partition test > Scylla Cloud is 1/7 the expense of DynamoDB when running equivalent workloads > Scylla Cloud: Average replication latency of 82ms.…

I was unwilling to sign up to read the actual benchmark report for the comparison to cockroachdb but it jumped out at me as odd. They solve completely different kinds of problems in my experience so I’m not surprised Scylla did better in raw throughout. That’s not interesting though. It would be just as weird for cockroach to put up a benchmark showing it outperforms in distributed sql queries. That said I’ve seen th…

They're technically both KV stores capable of offering you a tabular data model with multi-regional replication and leaderless horizontal scaling.

CockroachDB is focused on consistency with full SQL support while ScyllaDB focuses on availability with high-performance.

Re: Scylla – Real-Time Big Data Database

#22

Post by staticassertion » Tue, Aug 24, 2021, 11:21 PM UTC

jandrewrogers wrote: post #9

andrewstuart wrote: post #2

I posted this because I'm interested to hear from anyone using it - how has it worked out for you? I note it's written in C++ which is a bit of a surprise - I'd expected Rust or Golang. Interesting as well is is AGPL - licensing is always contentious: https://github.com/scylladb/scylla/blob/master/LICENSE.AGPL

If you are building a database engine that strongly prioritizes performance, and Scylla does position itself that way, then C++ is the only practical choice today for many people, depending on the details. It isn't that C++ is great, though modern versions are pretty nice, but that it wins by default. Garbage collected languages like Golang and high-performance database kernels are incompatible because the GC interfe…

> Rust has an issue unique to Rust in the specific case of high-performance database kernels. The internals of high-performance databases are full of structures, behaviors, and safety semantics that Rust's safety checking infrastructure is not designed to reason about. Consequently, to use Rust in a way that produces equivalent performance requires marking most of the address space as "unsafe". And while you could do this, Rust is currently less expressive than modern C++ for this type of code anyway, so it isn't ergonomic either.

None of that sounds right to me.

More likely the developers already know C++, there's already a lot of KV stores built in C++, and Rust is a relatively new player. Scylla was released in 2015, Rust hit 1.0 in 2015, seems obvious why Scylla didn't go with Rust.

edit: Yep, from further down

> So if we were starting at this point in time, I would take a hard look at Rust, and I imagine that we would pick it instead of C++. Of course, when we started Rust didn’t have the maturity that it has now, but it has progressed a long time since then and I’m following it with great interest. I think it’s a well-done language.

Re: Scylla – Real-Time Big Data Database

#23

Post by actually_a_dog » Tue, Aug 24, 2021, 11:24 PM UTC

Whenever I hear Scylla mentioned, I often wonder: where is Charybdis? They do work as a team, after all. :-)

Re: Scylla – Real-Time Big Data Database

#24

Post by masterof0 » Tue, Aug 24, 2021, 11:29 PM UTC

ethelward wrote: post #16

masterof0 wrote: post #12

Earlier quoted context omitted.

You read my mind. LOL. "Mr. Developer, can you please write your project in Rust, or __insert_your_meme_language_here__, or Javascript?"

Fromthe mouth of CockraochDB's CTO: ‶So if we were starting at this point in time, I would take a hard look at Rust, and I imagine that we would pick it instead of C++.″

It was a joke, to capture the sentiment here in HN. Rust is awesome, and most people know it. My point was that people will focus more often on which language is used, rather than the technical design, performance, etc...

Re: Scylla – Real-Time Big Data Database

#25

Post by PeterCorless » Tue, Aug 24, 2021, 11:30 PM UTC

actually_a_dog wrote: post #23

Whenever I hear Scylla mentioned, I often wonder: where is Charybdis? They do work as a team, after all. :-)

Charybdefs is our fault injection filesystem. Though it's based in Thrift, the ol' API that predated CQL.

https://github.com/scylladb/charybdefs

More modern is Project Circe, our efforts to make Scylla into an even more monstrous database:

https://www.scylladb.com/2021/01/12/making-scylla-a-monstrou...

Re: Scylla – Real-Time Big Data Database

#26

Post by [deleted] » Tue, Aug 24, 2021, 11:40 PM UTC

PeterCorless wrote: post #17

criticaltinker wrote: post #6

The benchmarks against DynamoDB, Bigtable, & CockroachDB [1] appear quite impressive - anyone have real world experience that can attest to these claims of improved performance and reduced cost? > Scylla vs DynamoDB – Database Benchmark > 20x better throughput in the hot-partition test > Scylla Cloud is 1/7 the expense of DynamoDB when running equivalent workloads > Scylla Cloud: Average replication latency of 82ms.…

We just posted this today. Latest Cassandra 4.0 vs. Scylla 4.4. Note: Cassandra 4.0 is a HUGE improvement over Cassandra 3.11. But we're still many times faster: https://www.scylladb.com/2021/08/24/apache-cassandra-4-0-vs-...

[deleted]

Re: Scylla – Real-Time Big Data Database

#27

Post by robmccoll » Tue, Aug 24, 2021, 11:51 PM UTC

enedil wrote: post #7

andrewstuart wrote: post #2

I posted this because I'm interested to hear from anyone using it - how has it worked out for you? I note it's written in C++ which is a bit of a surprise - I'd expected Rust or Golang. Interesting as well is is AGPL - licensing is always contentious: https://github.com/scylladb/scylla/blob/master/LICENSE.AGPL

Quoting the interview with ScyllaDB CTO, Avi Kivity ( https://www.scylladb.com/2020/06/30/ask-me-anything-with-avi... ) > Q: Would you implement Scylla in Go, Rust or Javascript if you could? > Avi: Good question. I wouldn’t implement Scylla in Javascript. It’s not really a high-performance language, but I will note that Node.js and Seastar share many characteristics. Both are using a reactor pattern and designed for…

I'd be careful with the idea of predictability and allocation. The best way to get predictabile performance is to avoid dynamic allocation altogether. The next best is to do your own allocation (slab-base per request, memory pools, etc.). General purpose dynamic memory management is a bin-packing problem (NP-hard).

Re: Scylla – Real-Time Big Data Database

#28

Post by robmccoll » Tue, Aug 24, 2021, 11:51 PM UTC

jandrewrogers wrote: post #9

andrewstuart wrote: post #2

I posted this because I'm interested to hear from anyone using it - how has it worked out for you? I note it's written in C++ which is a bit of a surprise - I'd expected Rust or Golang. Interesting as well is is AGPL - licensing is always contentious: https://github.com/scylladb/scylla/blob/master/LICENSE.AGPL

If you are building a database engine that strongly prioritizes performance, and Scylla does position itself that way, then C++ is the only practical choice today for many people, depending on the details. It isn't that C++ is great, though modern versions are pretty nice, but that it wins by default. Garbage collected languages like Golang and high-performance database kernels are incompatible because the GC interfe…

I wouldn't write off plain old C either.

Re: Scylla – Real-Time Big Data Database

#29

Post by nerpderp82 » Wed, Aug 25, 2021, 12:01 AM UTC

jandrewrogers wrote: post #9

andrewstuart wrote: post #2

I posted this because I'm interested to hear from anyone using it - how has it worked out for you? I note it's written in C++ which is a bit of a surprise - I'd expected Rust or Golang. Interesting as well is is AGPL - licensing is always contentious: https://github.com/scylladb/scylla/blob/master/LICENSE.AGPL

If you are building a database engine that strongly prioritizes performance, and Scylla does position itself that way, then C++ is the only practical choice today for many people, depending on the details. It isn't that C++ is great, though modern versions are pretty nice, but that it wins by default. Garbage collected languages like Golang and high-performance database kernels are incompatible because the GC interfe…

I always love your take even if I don't agree, SpaceCurve was a phenomenal system, one of the most pragmatic, high performance, easy to use MPP database systems I have ever used. We never met btw, was just a user.

But I think you are wrong about Rust not having the right machinery for making high performance dbs. Two examples are Noria and Materialize

https://github.com/mit-pdos/noria

and it its 50k lines, in the immediate codebase, there are 40 uses of unsafe.

In Materialize's 125k of Rust, there are 76 direct uses of unsafe.

https://github.com/MaterializeInc/materialize

Re: Scylla – Real-Time Big Data Database

#30

Post by RussianCow » Wed, Aug 25, 2021, 12:09 AM UTC

jandrewrogers wrote: post #9

andrewstuart wrote: post #2

I posted this because I'm interested to hear from anyone using it - how has it worked out for you? I note it's written in C++ which is a bit of a surprise - I'd expected Rust or Golang. Interesting as well is is AGPL - licensing is always contentious: https://github.com/scylladb/scylla/blob/master/LICENSE.AGPL

If you are building a database engine that strongly prioritizes performance, and Scylla does position itself that way, then C++ is the only practical choice today for many people, depending on the details. It isn't that C++ is great, though modern versions are pretty nice, but that it wins by default. Garbage collected languages like Golang and high-performance database kernels are incompatible because the GC interfe…

> Consequently, to use Rust in a way that produces equivalent performance requires marking most of the address space as "unsafe". And while you could do this, Rust is currently less expressive than modern C++ for this type of code anyway, so it isn't ergonomic either.

Based on my (admittedly limited) experience with Rust, this isn't true. Yes, you'd likely have to use "unsafe" a few times in order to implement a database system in Rust, but you would only need to do this for certain types of low-level data structures. The uses of those data structures—which would represent the majority of your code—would almost certainly be written in safe Rust. Don't throw the baby out with the bathwater.

I also contest the assertion that Rust is "less expressive" than C++; I have found Rust to be very expressive and concise for such a safe language. But I also don't have a ton of experience with either one, so don't take my word for that.

The real answer as to why Scylla does not use Rust is that the language simply wasn't very mature when they started. It also helps that there are significantly more engineers that know C++ than those that know Rust.