Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka vs. Redpanda performance – do the claims add up?

jack-vanlightly.com

Post reply on HN
Print view View original

21–30 of 147 posts

Re: Kafka vs. Redpanda performance – do the claims add up?

#21

Post by globalreset »

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety.

And then in this article it's explained how Kafka is actually unsafe:

> Kafka may handle simultaneous broker crashes but simultaneous power failure is a problem.

just against simultaneous node crashes (whole VM/machine).

I mean - sure in practice running in different AZs, etc. will probably be good enough, but technically...

Re: Kafka vs. Redpanda performance – do the claims add up?

#22

Post by agallego »

alex here, original author of redpanda

is hard to respond to a 6-part blog series content - released all at once - on an HN thread.

- what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here.

- the kafka partition model of one segment per partition could be optimized in both arch

- the benefit for all of us, is that all of these things will be committed to the OMB (open messaging benchmark) and will be on git for anyone interested in running it themselves.

- we welcome all confluent customers (since the post is from the field cto office) to benchmark against us and choose the best platform. this is how engineering is done. In fact, we will help you run it for you at no cost. Your hardware, your workload head-to-head. We'll help you set it up with both.... but let's keep the rest of the thread technical.

- log.flush.interval.messages=1 - this is something we've taken a stance a long long time ago in 2019. As someone who has personally talked to hundreds of enterprises to date, most workloads in the world should err on the side of safety and flushing to disk (fsync()). Hardware is very good today and you no longer have to choose between safety and reasonable performance. This isn't the high latency you used to see on spinning disks.

Re: Kafka vs. Redpanda performance – do the claims add up?

#23

Post by jvanlightly »

purpleblue wrote: post #18

TLDR: "I work at Confluent, the owners of Kafka, and I have determined through my tests that Redpanda's performance is greatly exaggerated." I don't think we can get a less reliable or trustworthy set of performance tests than when someone's paycheck depends on the outcome of those tests. If Redpanda's performance were found to be better, would he really publish the test results?

Author here. Anyone can run these tests. It's available for anyone to run and check my results.

Re: Kafka vs. Redpanda performance – do the claims add up?

#24

Post by dilyevsky »

globalreset wrote: post #21

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. And then in this article it's explained how Kafka is actually unsafe: > Kafka may h…

Yep kafka by default is setup to lose data, many people dont know or dont care it seems…

Re: Kafka vs. Redpanda performance – do the claims add up?

#25

Post by notfromhere »

insanitybit wrote: post #19
purpleblue wrote: post #18

TLDR: "I work at Confluent, the owners of Kafka, and I have determined through my tests that Redpanda's performance is greatly exaggerated." I don't think we can get a less reliable or trustworthy set of performance tests than when someone's paycheck depends on the outcome of those tests. If Redpanda's performance were found to be better, would he really publish the test results?

I mean, the other benchmarks we have are from RedPanda, so we're comparing one biased set of benchmarks to another biased set of benchmarks. Ultimately it's a matter of the reader understanding the methodology and drawing their own conclusions based on their own experience. I appreciate that the author explains the changes they've made, the impact of those changes, and why they think the changes are reasonable (ex: d…

Disabling fsync is dubious.

I do find it interesting that Confluent feels the need to respond to RP given the disparities in size, install base, etc.

Re: Kafka vs. Redpanda performance – do the claims add up?

#26

Post by [deleted] »

pram wrote: post #7
victor106 wrote: post #5

Earlier quoted context omitted.

I agree with Confluent pricing, we had the same experience. We switched to pub/sub and Azure Event hubs. I don't even understand why Confluent should price their offering so high. ITs not like Real time is an exclusive service that other platforms don't have.

Because they have stock they need to pump. I've found talking to Confluent about anything is a complete waste of time unless it's a very specific technical issue. They're always pushing their cloud as the solution, and it's very aggressive.

[deleted]

Re: Kafka vs. Redpanda performance – do the claims add up?

#27

Post by jvanlightly »

dilyevsky wrote: post #24
globalreset wrote: post #21

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. And then in this article it's explained how Kafka is actually unsafe: > Kafka may h…

Yep kafka by default is setup to lose data, many people dont know or dont care it seems…

Well, that just isn't accurate really. Kafka would need simulteanous VM failure across all AZs. That just doesn't happen in the real world often enough to worry about. It has never happened in Confluent Cloud. RP have a similar issue. Single AZ deployments with local NVMe drives. AZ loses power, a majority of brokers could lose all their data. Then there's data corruption. Fsyncs alone don't save you. The next step would be to implement Protocol Aware Recovery (https://www.usenix.org/conference/fast18/presentation/alagap...) like TigerBeetle have. Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that?

Re: Kafka vs. Redpanda performance – do the claims add up?

#28

Post by minhazm »

purpleblue wrote: post #18

TLDR: "I work at Confluent, the owners of Kafka, and I have determined through my tests that Redpanda's performance is greatly exaggerated." I don't think we can get a less reliable or trustworthy set of performance tests than when someone's paycheck depends on the outcome of those tests. If Redpanda's performance were found to be better, would he really publish the test results?

I actually enjoy these kinds of benchmarks. They're both incentivized to show their own platforms running in the most optimal setups and they're also incentivized to call out any BS from the other party. In the end users get to see the good and the bad of both platforms.

For this particular post I like that they explained each settings change they're making and why. In many of these benchmarks people will make some change and either not mention it or won't explain why they made the change and users are left trying to figure it out.

Re: Kafka vs. Redpanda performance – do the claims add up?

#29

Post by dilyevsky »

jvanlightly wrote: post #27
dilyevsky wrote: post #24

Earlier quoted context omitted.

Yep kafka by default is setup to lose data, many people dont know or dont care it seems…

Well, that just isn't accurate really. Kafka would need simulteanous VM failure across all AZs. That just doesn't happen in the real world often enough to worry about. It has never happened in Confluent Cloud. RP have a similar issue. Single AZ deployments with local NVMe drives. AZ loses power, a majority of brokers could lose all their data. Then there's data corruption. Fsyncs alone don't save you. The next step w…

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss.

> Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that?

Sure, why not? I think zk doesn’t do fsync too btw

Re: Kafka vs. Redpanda performance – do the claims add up?

#30

Post by morelisp »

notfromhere wrote: post #25
insanitybit wrote: post #19

Earlier quoted context omitted.

I mean, the other benchmarks we have are from RedPanda, so we're comparing one biased set of benchmarks to another biased set of benchmarks. Ultimately it's a matter of the reader understanding the methodology and drawing their own conclusions based on their own experience. I appreciate that the author explains the changes they've made, the impact of those changes, and why they think the changes are reasonable (ex: d…

Disabling fsync is dubious. I do find it interesting that Confluent feels the need to respond to RP given the disparities in size, install base, etc.

I've been watching Redpanda for a couple years primarily because I'm interested in their wasm data transformations. In the past 3 months I've heard it mentioned several dozen times by other teams in our company, vs. maybe 2-3 times in the >1y prior. So something seems in the air, and presumably Confluent has noticed.

I'm not sure why, Kafka per se doesn't seem to have really dropped any significant balls lately (and we're self-hosted so Confluent isn't very relevant).

Post reply on HN