Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

scylladb.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#91

Post by pron »

acconsta wrote: post #90
pron wrote: post #76

Earlier quoted context omitted.

I think that's a generalization that simply shifts the burden elsewhere, and cannot be said to be "the right" architecture in general. There is a reason CPUs implement cache-coherence on top of their "innate" shared-nothing design, and the reason is abstraction. If you don't need certain abstractions, then a sharded approach is indeed optimal, but if you do, then you have to implement them at some level or another, a…

You can still offer isolated transactions, tunable consistency, etc. within a shard though, which Cassandra does. And yes, you can write high performance Java, but for whatever reasons the Cassandra codebase isn't an example of that. They just did a big storage engine rewrite and the result is slower . https://issues.apache.org/jira/browse/CASSANDRA-7486

> You can still offer isolated transactions, tunable consistency, etc. within a shard though

And if that happens to be exactly all you need then that's great! :)

> And yes, you can write high performance Java, but for whatever reasons the Cassandra codebase isn't an example of that.

I don't know anything about the Cassandra codebase, but one thing I'm often asked is if you're not going to use the GC in Java, why write Java at all. The answer is that the very tight core code like a shard event loop turns out to be a rather small part of the total codebase. There's more code dedicated to support functions (such as monitoring and management) than the core, and relying on the GC for that makes writing all that code much more convenient and doesn't affect your performance.

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#92

Post by acconsta »

pron wrote: post #91
acconsta wrote: post #90

Earlier quoted context omitted.

You can still offer isolated transactions, tunable consistency, etc. within a shard though, which Cassandra does. And yes, you can write high performance Java, but for whatever reasons the Cassandra codebase isn't an example of that. They just did a big storage engine rewrite and the result is slower . https://issues.apache.org/jira/browse/CASSANDRA-7486

> You can still offer isolated transactions, tunable consistency, etc. within a shard though And if that happens to be exactly all you need then that's great! :) > And yes, you can write high performance Java, but for whatever reasons the Cassandra codebase isn't an example of that. I don't know anything about the Cassandra codebase, but one thing I'm often asked is if you're not going to use the GC in Java, why writ…

Google built multi-row transactions on top of a partitioned row store though? I guess I'm not really sure which applications have a shared memory architecture like the one you're describing.

And to be fair, some people are pretty productive in modern C++. It's a shame the JNI isn't better so you can have the best of both worlds.

(for the record, Quasar is #1 on my list of libraries to try if I go back to Java)

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#93

Post by acconsta »

a8da6b0c91d wrote: post #88
jandrewrogers wrote: post #24

Earlier quoted context omitted.

Over the last decade, the distributed system nature of modern server hardware internals has become painfully evident in how software architectures scale on a single machine. The traditional approaches -- multithreading, locking, lock-free structures, etc -- are all forms of coordination and agreement in a distributed system, with the attendant scalability problems if not used very carefully. At some point several yea…

> one process per core, each locked to a single core; use locked local RAM only (effectively limiting NUMA); direct dedicated network queue (bypass kernel); direct storage I/O (bypass kernel) I have no idea how to do any of these things. What are the system/api calls to lock a process to a kernel? How do you bypass kernel IO?

Take a look at sched_setaffinity and Intel DPDK.

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#94

Post by pron »

acconsta wrote: post #92
pron wrote: post #91

Earlier quoted context omitted.

> You can still offer isolated transactions, tunable consistency, etc. within a shard though And if that happens to be exactly all you need then that's great! :) > And yes, you can write high performance Java, but for whatever reasons the Cassandra codebase isn't an example of that. I don't know anything about the Cassandra codebase, but one thing I'm often asked is if you're not going to use the GC in Java, why writ…

Google built multi-row transactions on top of a partitioned row store though? I guess I'm not really sure which applications have a shared memory architecture like the one you're describing. And to be fair, some people are pretty productive in modern C++. It's a shame the JNI isn't better so you can have the best of both worlds. (for the record, Quasar is #1 on my list of libraries to try if I go back to Java)

Of course you can implement any shared/consistent memory transaction on top of shards -- after all, the CPU implements shared memory on top of message-passing in a shared-nothing architecture, too. It's just that then you end up implementing that abstraction yourself. If you need it, someone has to implement it, and if you need it at the machine level, it's better to rely on its hardware implementation than re-implement the same thing in software. Naive implementations end up creating much more contention (i.e. slow communications) than a sophisticated use of hardware concurrency (i.e. communication) instructions.

My point is that if you're providing a shared-memory abstraction to your user (like arbitrary transactions) -- even at a very high level -- then your architecture isn't "shared-nothing", period. Somewhere in your stack there's an implementation of a shared-memory abstraction. And if you decide to call anything that doesn't use shared memory at CPU/OS level "shared nothing", then that's an arbitrary and rather senseless distinction, because even at the CPU/OS level, shared memory is implemented on top of message-passing. So the cost of a shared abstraction is incurred when it's provided to the user, and is completely independent of how it's implemented. The only way to avoid it is to restrict the programming model and not provide the abstraction. If doing that is fine for the user -- great, but there's no way to have this abstraction without paying for it.

And JNI is better now[1] (I've used JNR to implement FUSE filesystems in pure Java). JNR will serve as the basis for "JNI 2.0" -- Project Panama[2]. And thanks!

[1]: https://github.com/jnr/jnr-ffi

[2]: http://openjdk.java.net/projects/panama/

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#95

Post by _Codemonkeyism »

ryeguy wrote: post #59
_Codemonkeyism wrote: post #25

Earlier quoted context omitted.

10x speedup (same algorithms, same architecture) replacing Java with C++ is not possible (~2x at max). One of the latest benchmarks I've seen is "Comparison of Programming Languages in Economics" [1] for code without any IO just number crunching, has a 1.91 to 2.69 speedup of using C++ compared to Java. So any code involving IO is going to be slower. Replacing bad Java code with excellent machine aligned C++ a 10x sp…

You are placing way too much weight in microbenchmarks. You simply can't use them to make a sweeping statement like you just did. Writing code that is identical to one another from language to language is not idiomatic and is not representative of how you would write each in a large scale project such as cassandra. Java has a ton of overhead that C++ doesn't. Each object has metadata which results in more "cold data"…

1. I don't think that the number crunching paper I've cited is a "microbenchmark".

2. You seem to have missed the "bad Java" part, and my reply to Mechanical Sympathy.

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#96

Post by cbsmith »

glommer wrote: post #50
cbsmith wrote: post #26

Earlier quoted context omitted.

It's particularly flawed given: a) IO is such a large portion of the problem b) Hypertable isn't just way, way faster.

IO is not only a large part. It is the main part. That is why it is important to get it right : scylla for instance does not leave the cache to the OS. It has its own caches for everything. Never blocks on IO or page faults because all IO bypasses the kernel. And those are just two tiny examples.

Even on the networking side, you can see from projects like this that you can get what should be enough messaging performance for any NoSQL store out of Java: https://github.com/real-logic/Aeron

Their Java throughput is about 70% of their C++11 throughput, and that's with a pretty synthetic benchmark where there is not any logic behind those messages. Once you add in some real logic there, it gets even thinner.

They aren't doing user space networking, but that actually ought to allow Java to do even better.

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#97

Post by acconsta »

pron wrote: post #94
acconsta wrote: post #92

Earlier quoted context omitted.

Google built multi-row transactions on top of a partitioned row store though? I guess I'm not really sure which applications have a shared memory architecture like the one you're describing. And to be fair, some people are pretty productive in modern C++. It's a shame the JNI isn't better so you can have the best of both worlds. (for the record, Quasar is #1 on my list of libraries to try if I go back to Java)

Of course you can implement any shared/consistent memory transaction on top of shards -- after all, the CPU implements shared memory on top of message-passing in a shared-nothing architecture, too. It's just that then you end up implementing that abstraction yourself. If you need it, someone has to implement it, and if you need it at the machine level, it's better to rely on its hardware implementation than re-implem…

Very interesting, thanks

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#98

Post by glommer »

cbsmith wrote: post #66
glommer wrote: post #50

Earlier quoted context omitted.

IO is not only a large part. It is the main part. That is why it is important to get it right : scylla for instance does not leave the cache to the OS. It has its own caches for everything. Never blocks on IO or page faults because all IO bypasses the kernel. And those are just two tiny examples.

> scylla for instance does not leave the cache to the OS. It has its own caches for everything Uh-huh... that's all pretty common for databases. Cassandra would fit that description. > Never blocks on IO or page faults because all IO bypasses the kernel. That just seems nonsensical. Sometimes, you are waiting for IO. That's just reality. It is conceivable you bypass the kernel for I/O, but that creates a lot of compl…

Cassandra has a row cache that it does not necessarily use. Most of its data sits in the linux page cache. Because the SSTables are mmaped into memory, you can't get rid of that: even if you do use the row cache, you would be at best using twice as much cache memory.

Scylla never touches the page cache. All IO in seastar is direct IO and then scylla caches everything itself. We always know when the disk access is going to happen. The OS paging mechanism does not do a thing.

As for waiting for IO, of course IO does not complete immediately. But you can either block and wait for it, as Cassandra does (it doesn't even have the option not to in the case of the mmaped regions) or you can do something fully async like seastar that guarantees you never block waiting for IO.

Re: ScyllaDB: Drop-in replacement for Cassandra that claims to be 10x faster

#99

Post by infinite8s »

jandrewrogers wrote: post #24
acconsta wrote: post #18

Earlier quoted context omitted.

Agreed, but which architectural features are you referring to?

Over the last decade, the distributed system nature of modern server hardware internals has become painfully evident in how software architectures scale on a single machine. The traditional approaches -- multithreading, locking, lock-free structures, etc -- are all forms of coordination and agreement in a distributed system, with the attendant scalability problems if not used very carefully. At some point several yea…

This approach seems to dovetail nicely with unikernel approaches. Have you experimented with combining the above constraints with a system like MirageOS?
Post reply on HN