Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thread-Per-Core Buffer Management for a modern storage system

vectorized.io

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Thread-Per-Core Buffer Management for a modern storage system

#11

Post by agallego »

user5994461 wrote: post #10
agallego wrote: post #7

Earlier quoted context omitted.

Indeed. I think you have different saturation points the wider the use cases you hit. One example w/ a single-core (which btw, agreed whole heartedly for io) is checksumming + decoding. For kafka, we have multiple indexes - a time index and an offset index which are simple metadata. the trouble becomes on how you handle decompression+checksumming+compression for supporting compacted topics. ( https://github.com/vecto…

>>> the trouble becomes on how you handle decompression+checksumming+compression gzip will cap 1 MB/s with the strongest compression setting and 50 MB/s with the fastest setting, which is really slow. The first step to improve kafka is for kafka to adopt zstd compression. Another thing that really hurts is SSL. Desktop CPU with AES instructions can push 1 GB/s so it's not too bad, but that may not the the CPU you hav…

Kafka has `zstd` encoding.

Here is our version of the streaming decoder i wrote a while ago https://github.com/vectorizedio/redpanda/blob/dev/src/v/comp...

that's our default for our internal RPC as well.

in fact kafka protocol support lz4, zstd, snappy, gzip all of them. and you can change them per batch. compression is good w/ kafka.

Re: Thread-Per-Core Buffer Management for a modern storage system

#12

Post by dotnwat »

monstrado wrote: post #9
dotnwat wrote: post #4

Noah here, developer at Vectorized. Happy to answer any questions.

What kind of gaps are there currently between Kafka and RedPanda? Particularly in the "Enterprise" world (e.g. security, etc).

In progress right now are transactions and ACLs. A preliminary form of ACLs with SCRAM will be available later this month. Transactions will come early next year. Those are probably the most visible differences.

Re: Thread-Per-Core Buffer Management for a modern storage system

#13

Post by loeg »

user5994461 wrote: post #10
agallego wrote: post #7

Earlier quoted context omitted.

Indeed. I think you have different saturation points the wider the use cases you hit. One example w/ a single-core (which btw, agreed whole heartedly for io) is checksumming + decoding. For kafka, we have multiple indexes - a time index and an offset index which are simple metadata. the trouble becomes on how you handle decompression+checksumming+compression for supporting compacted topics. ( https://github.com/vecto…

>>> the trouble becomes on how you handle decompression+checksumming+compression gzip will cap 1 MB/s with the strongest compression setting and 50 MB/s with the fastest setting, which is really slow. The first step to improve kafka is for kafka to adopt zstd compression. Another thing that really hurts is SSL. Desktop CPU with AES instructions can push 1 GB/s so it's not too bad, but that may not the the CPU you hav…

lz4 is a good option for really high-performance compression as well. (Zstd is my general recommmendation, and both beat the pants off of gzip, but for very high throughput applications lz4 still beats zstd. Both are designs from Yann Collet.)

Re: Thread-Per-Core Buffer Management for a modern storage system

#14

Post by monstrado »

dotnwat wrote: post #12
monstrado wrote: post #9

Earlier quoted context omitted.

What kind of gaps are there currently between Kafka and RedPanda? Particularly in the "Enterprise" world (e.g. security, etc).

In progress right now are transactions and ACLs. A preliminary form of ACLs with SCRAM will be available later this month. Transactions will come early next year. Those are probably the most visible differences.

That's great! Would it be correct to assume that KSQL works?

Re: Thread-Per-Core Buffer Management for a modern storage system

#15

Post by dotnwat »

monstrado wrote: post #14
dotnwat wrote: post #12

Earlier quoted context omitted.

In progress right now are transactions and ACLs. A preliminary form of ACLs with SCRAM will be available later this month. Transactions will come early next year. Those are probably the most visible differences.

That's great! Would it be correct to assume that KSQL works?

Having not tried, I would expect ksql to not work until transaction support lands. That said, perhaps there are some ways to configure it to avoid dependencies on those underlying APIs.

Re: Thread-Per-Core Buffer Management for a modern storage system

#16

Post by agallego »

loeg wrote: post #13
user5994461 wrote: post #10

Earlier quoted context omitted.

>>> the trouble becomes on how you handle decompression+checksumming+compression gzip will cap 1 MB/s with the strongest compression setting and 50 MB/s with the fastest setting, which is really slow. The first step to improve kafka is for kafka to adopt zstd compression. Another thing that really hurts is SSL. Desktop CPU with AES instructions can push 1 GB/s so it's not too bad, but that may not the the CPU you hav…

lz4 is a good option for really high-performance compression as well. (Zstd is my general recommmendation, and both beat the pants off of gzip, but for very high throughput applications lz4 still beats zstd. Both are designs from Yann Collet.)

indeed. though, the recent zstd changes w/ different levels of compression sort of close the gap in perf that lz4 had over zstd. (if interested in this kind of detail for a new streaming storage engiene, i gave a talk last week at the facebook performance summit - https://twitter.com/perfsummit1/status/1337603028677902336)

Re: Thread-Per-Core Buffer Management for a modern storage system

#18

Post by agallego »

mirekrusin wrote: post #17

What is the point of talking performance-by-thread-per-core if raft sits in front of it, ie. only one will do the work at any time anyway?

so redpanda partitions 'raft' groups per kafka partition. so in the `topic/partition` model every partition is it's own raft group (similar to multi raft in cockroachdb). So it is in fact even more important due to the replication cost and therefore the additional work of checksumming, compression, etc.

Last, a coordinator core for the one of the TCP connections from a client will likely make requests to remote cores (say you receive a request on core 44, but the destination is core 66), so having a thread per core with explicit message passing is pretty fundamental.

    ss::future>
    dispatch_hbeats_to_core(ss::shard_id shard, hbeats_ptr requests) {
        return with_scheduling_group(
          get_scheduling_group(),
          [this, shard, r = std::move(requests)]() mutable {
              return _group_manager.invoke_on(
                shard,
                get_smp_service_group(),
                [this, r = std::move(r)](ConsensusManager& m) mutable {
                    return dispatch_hbeats_to_groups(m, std::move(r));
                });
          });
    }
Here is some code that shows importance of accounting the x-core comms explicitly

Re: Thread-Per-Core Buffer Management for a modern storage system

#19

Post by mirekrusin »

agallego wrote: post #18
mirekrusin wrote: post #17

What is the point of talking performance-by-thread-per-core if raft sits in front of it, ie. only one will do the work at any time anyway?

so redpanda partitions 'raft' groups per kafka partition. so in the `topic/partition` model every partition is it's own raft group (similar to multi raft in cockroachdb). So it is in fact even more important due to the replication cost and therefore the additional work of checksumming, compression, etc. Last, a coordinator core for the one of the TCP connections from a client will likely make requests to remote cores…

Ok, thanks. Does redpanda do some kind of auto anti-affinity on hosts for partition group to spread across remote cores?

ps. redpanda link from article is broken, goes to https://vectorized.io/blog/tpc-buffers/vectorized.io/redpand... 404

Post reply on HN