Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka vs. Redpanda performance – do the claims add up?

jack-vanlightly.com

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Kafka vs. Redpanda performance – do the claims add up?

#61

Post by uberduper »

globalreset wrote: post #21

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. And then in this article it's explained how Kafka is actually unsafe: > Kafka may h…

If your workload demands it, then by all means set log.flush.interval.messages=1 or find an alternative solution that is a better match for your requirements.

Kafka has never pretended that ack'd messages have been persisted to disk, only that they've been replicated per your requested acks.

Re: Kafka vs. Redpanda performance – do the claims add up?

#62

Post by nezirus »

skyde wrote: post #49
nemothekid wrote: post #40

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…

Exactly I will never ever try MongoDB because of that. A database that do not fsync should not be called a database.

Kafka doesn't do any stupid tricks, but uses the underlying platform for the full potential: https://kafka.apache.org/documentation/#linuxflush

With the usual recommended settings, XFS filesystem, 3 replicas, 2 "in-sync" replicas, etc., it is rather safe. You can also tune background flush to your liking.

The above tradeoffs are very reasonable and Kafka runs very fast on slow disk s(magnetic or in cloud), and even faster on SSD/NVMe disks.

Re: Kafka vs. Redpanda performance – do the claims add up?

#63

Post by kevin_nisbet »

deschutes wrote: post #48
globalreset wrote: post #21

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. And then in this article it's explained how Kafka is actually unsafe: > Kafka may h…

You can't eliminate the risk of data loss, only control for it. fsync is one such control. Empirically, having separate power failure domains strongly controls for the power loss risk. In the tail there are all kinds of things that will lose you data. I've actually seen systems lose data with the fsync every message strategy on simultaneous power loss. There was latent corruption of the filesystem due to a kernel bug…

I only remember losing one etcd cluster, and it was due to something along these lines. Data center at the customer site lost power, and we were called when they couldn't recover our software. All the etcd volumes were corrupted, and after volume recovery by the customer IT department, we found all our etcd files corrupted.

My best guess is their volume systems simply lied about the fsync, which I've heard of a few times about different vendors.

Re: Kafka vs. Redpanda performance – do the claims add up?

#64

Post by nemothekid »

haggy wrote: post #59
nemothekid wrote: post #40

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…

Kafka, unlike Mongo DB, relies on recovery/replication instead of fsync: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... Kafka has never tried to hide that fact and it does not, in any way, make Kafka unsafe.

I don't think Kafka using eschewing fsyncs is a bad thing; I'm aware of the risks. What I'm pointing out, and what got Mongo killed in the court of public opinion, was saying "our database is blazing fast because we turned off fsyncs".

Benchmarking a system that fsyncs every write to one that doesn't isn't an apples-to-apples comparison. You are free to make the argument that you might not need them, but if you are benchmarking systems and one of them fsyncs by default, that is the level of durability I'm going to expect, otherwise I can assume the other guy will be just as fast if he turns off fsyncs as well.

Re: Kafka vs. Redpanda performance – do the claims add up?

#65

Post by uberduper »

purpleblue wrote: post #18

TLDR: "I work at Confluent, the owners of Kafka, and I have determined through my tests that Redpanda's performance is greatly exaggerated." I don't think we can get a less reliable or trustworthy set of performance tests than when someone's paycheck depends on the outcome of those tests. If Redpanda's performance were found to be better, would he really publish the test results?

Apache Software Foundation owns Kafka.

Re: Kafka vs. Redpanda performance – do the claims add up?

#66

Post by sitkack »

I'd like to see a baseline of fio and iperf3 for these same instances so we know how much raw performance is available for disk, network alone and together.

Cloud instances have their own performance pathologies, esp in the use of remote disks.

As for RP and Kafka performance, I'd love to see a parameter sweep over both configuration dimensions as well as workload. I know this is a large space, but it needs to be done to characterize the available capacity, latency and bandwidth.

Re: Kafka vs. Redpanda performance – do the claims add up?

#67

Post by spenczar5 »

insanitybit wrote: post #51
Spivak wrote: post #46

Earlier quoted context omitted.

> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every…

> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.

Then enable fsync. I don't really see a way around requiring synchronization to persistent disk if you want persistence cross power outages, right?

Re: Kafka vs. Redpanda performance – do the claims add up?

#68

Post by purpleblue »

uberduper wrote: post #65
purpleblue wrote: post #18

TLDR: "I work at Confluent, the owners of Kafka, and I have determined through my tests that Redpanda's performance is greatly exaggerated." I don't think we can get a less reliable or trustworthy set of performance tests than when someone's paycheck depends on the outcome of those tests. If Redpanda's performance were found to be better, would he really publish the test results?

Apache Software Foundation owns Kafka.

Meh. It's obvious Confluent exploited the status of being an Apache open source project in order to say they were open-source. But look at the make up of the PMC of Kafka and it's completely dominated by Confluent employees or former employees. Nothing gets done without Confluent's approval or best interest at heart.

Re: Kafka vs. Redpanda performance – do the claims add up?

#69

Post by fulafel »

haggy wrote: post #59
nemothekid wrote: post #40

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…

Kafka, unlike Mongo DB, relies on recovery/replication instead of fsync: https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-... Kafka has never tried to hide that fact and it does not, in any way, make Kafka unsafe.

Is durability preserved when you lose replica connectivity around the same time as power to your CPU? As tends to happen.

Re: Kafka vs. Redpanda performance – do the claims add up?

#70

Post by agallego »

llama052 wrote: post #60

We really wanted to try redpanda, but operationally it does not appear to be very k8s* native and infact looks like a lot of one off hand holding to get it working properly. Hopefully that can get ironed out in the future. Until then we will stick with the Strimzi operator and kafka. Also Confluent is absolutely pricing themselves out of the market. We looked at their self hosted confluent operator and they wanted so…

our real storage is s3 - local disk is for staging/raft layer. how is that not cloud native. if you are referring to cloud native as k8s it is true that our k8s operator was built mostly for our cloud but we released it... the good news is a new interface (same code) w/ more friendly user-defaults is about to get released. you can track it all on github tho.
Post reply on HN