Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka vs. Redpanda performance – do the claims add up?

jack-vanlightly.com

Post reply on HN
Print view View original

141–147 of 147 posts

Re: Kafka vs. Redpanda performance – do the claims add up?

#141

Post by zerd »

dilyevsky wrote: post #29
jvanlightly wrote: post #27

Earlier quoted context omitted.

Well, that just isn't accurate really. Kafka would need simulteanous VM failure across all AZs. That just doesn't happen in the real world often enough to worry about. It has never happened in Confluent Cloud. RP have a similar issue. Single AZ deployments with local NVMe drives. AZ loses power, a majority of brokers could lose all their data. Then there's data corruption. Fsyncs alone don't save you. The next step w…

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw

Zookeeper absolutely does fsync, you can't disable it (without libeatmydata). It will log a warning if fsync is too slow as well.

Re: Kafka vs. Redpanda performance – do the claims add up?

#142

Post by morelisp »

BeeOnRope wrote: post #139
dangoodmanUT wrote: post #102

This was a nice read! There are a few issues on both sides, some that others have mentioned and some that I have not seen yet: For Redpanda: 1. I don't like that they did not include full disk performance, not sure if that was intentional but it feels like it... Seems like and obvious gap in their testing. Perhaps most of their workloads have records time out rather than get pushed out by bytes first, not sure. 2. Th…

> Batching should be done on the client side anyway, as most packages already do by default. If you are worried about too many fsyncs degrading performance, batch harder on your clients. It's the better way to batch anyway. This is of course why performance suffers with 50 producers and 288 partitions: not because there is any inherent scale issue in supporting 50 clients (Repanda supports 1000s of clients), but beca…

Of course, have your producers linger is just another potential source of data loss if the client node dies before it can actually produce.

Re: Kafka vs. Redpanda performance – do the claims add up?

#143

Post by BeeOnRope »

morelisp wrote: post #142
BeeOnRope wrote: post #139

Earlier quoted context omitted.

> Batching should be done on the client side anyway, as most packages already do by default. If you are worried about too many fsyncs degrading performance, batch harder on your clients. It's the better way to batch anyway. This is of course why performance suffers with 50 producers and 288 partitions: not because there is any inherent scale issue in supporting 50 clients (Repanda supports 1000s of clients), but beca…

Of course, have your producers linger is just another potential source of data loss if the client node dies before it can actually produce.

This is not data loss sense we talk about for Kafka or other queues, however, since the messages have not been acked: the state of unacked messages is completely unknown and no guarantees are made about them.

Re: Kafka vs. Redpanda performance – do the claims add up?

#144

Post by morelisp »

BeeOnRope wrote: post #143
morelisp wrote: post #142

Earlier quoted context omitted.

Of course, have your producers linger is just another potential source of data loss if the client node dies before it can actually produce.

This is not data loss sense we talk about for Kafka or other queues, however, since the messages have not been acked: the state of unacked messages is completely unknown and no guarantees are made about them.

Again, if we’re talking about a full failure across all AZs, this feels like a distinction without a difference.

Re: Kafka vs. Redpanda performance – do the claims add up?

#145

Post by chaotic-good »

cortesoft wrote: post #4

> Redpanda end-to-end latency of their 1 GB/s benchmark increased by a large amount once the brokers reached their data retention limit and started deleting segment files. Current benchmarks are based on empty drive performance. This seems really disingenuous to use empty drive performance, since anyone who cares about performance is going to be caring about continuous use.

You have to provision you disk space accordingly. NVMe needs some free space to have good performance. In this case I think that in Redpanda benchmarks the disk space was available and in case of benchmarks done by Confluent guy the system was provisioned to use all disk space.

With page cache it's OK, because the FTL layer of the drive will work with 32MiB blocks but in case of Redpanda the drive will struggle because FTL mappings are complex and GC has more work. If Kafka would be doing fsync's the behaviour would be the same.

Overall, this looks like a smearing campaign against Redpanda. The guy who wrote this article works for Confluent and he published it on his own domain to look more neutral. The benchmarks are not fair because one of the systems is doing fsyncs and the other does not. Most differences could be explained by this fact alone.

Re: Kafka vs. Redpanda performance – do the claims add up?

#146

Post by chaotic-good »

comet-engine wrote: post #12
cortesoft wrote: post #4

> Redpanda end-to-end latency of their 1 GB/s benchmark increased by a large amount once the brokers reached their data retention limit and started deleting segment files. Current benchmarks are based on empty drive performance. This seems really disingenuous to use empty drive performance, since anyone who cares about performance is going to be caring about continuous use.

It's pretty ironic considering they blame JVM garbage collection for bad latency, but ignore their own disk garbage collection that also seems to cause some pretty bad latency.

The disk thrashes because of fsyncs (Kafka doesn't perform any fsync's). But you can provision more disk space to mitigate this problem. And it looks like the test was set up this way to make Redpanda look worse.

Re: Kafka vs. Redpanda performance – do the claims add up?

#147

Post by BeeOnRope »

morelisp wrote: post #144
BeeOnRope wrote: post #143

Earlier quoted context omitted.

This is not data loss sense we talk about for Kafka or other queues, however, since the messages have not been acked: the state of unacked messages is completely unknown and no guarantees are made about them.

Again, if we’re talking about a full failure across all AZs, this feels like a distinction without a difference.

We weren't necessarily talking about that at all but whether data "lost" because a client crashed before it received acknowledgement of a durable write from the server is somehow the same as losing data that has been acknowledged by the backend.

I argue they are not at all the same: it is, for example, the difference between getting an error when you try to place an online order and getting a successful confirmation but the order is then silently lost.

Post reply on HN