Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka vs. Redpanda performance – do the claims add up?

jack-vanlightly.com

Post reply on HN
Print view View original

41–50 of 147 posts

Re: Kafka vs. Redpanda performance – do the claims add up?

#41

Post by jakewins »

ceencee wrote: post #33
dilyevsky wrote: post #29

Earlier quoted context omitted.

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw

Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t a thing in the cloud at least. There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary. Data loss in kafka isn't a real problem that hurts real world users at all.

I was grabbing beer with a buddy who has ran some large - petabytes per month - Kafka deployments, and his experience was very much that Kafka will lose acked writes if not very carefully configured. He had direct experiences with data loss from JVM GC creating terrible flaky cluster conditions and, separately, from running out of disk on one cluster machine

Re: Kafka vs. Redpanda performance – do the claims add up?

#42

Post by fulafel »

A tangent but how do distributed robustness properties in face of communication hiccups compare between Redpanda and Kafka? Eg with Raft apparently you can still fail in presence of asymmetric network failures (like in https://blog.cloudflare.com/a-byzantine-failure-in-the-real-...)

Re: Kafka vs. Redpanda performance – do the claims add up?

#43

Post by nemothekid »

ceencee wrote: post #33
dilyevsky wrote: post #29

Earlier quoted context omitted.

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw

Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t a thing in the cloud at least. There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary. Data loss in kafka isn't a real problem that hurts real world users at all.

I can't list names about the "unserious" people who aren't running multi-AZ, but this is the approach to durability that MongoDB took ~15 years ago and they have never lived it down.

It may just be that data reliability isn't a huge concern for messaging queues, so it's less of an issue, but pretending the risk isn't there doesn't help anyone.

Re: Kafka vs. Redpanda performance – do the claims add up?

#44

Post by insanitybit »

morelisp wrote: post #30
notfromhere wrote: post #25

Earlier quoted context omitted.

Disabling fsync is dubious. I do find it interesting that Confluent feels the need to respond to RP given the disparities in size, install base, etc.

I've been watching Redpanda for a couple years primarily because I'm interested in their wasm data transformations. In the past 3 months I've heard it mentioned several dozen times by other teams in our company, vs. maybe 2-3 times in the >1y prior. So something seems in the air, and presumably Confluent has noticed. I'm not sure why, Kafka per se doesn't seem to have really dropped any significant balls lately (and…

Everyone is thinking about their cloud costs right now, so something that offers higher perf and lower ops is going to be more relevant today.

Re: Kafka vs. Redpanda performance – do the claims add up?

#45

Post by whartung »

purpleblue wrote: post #18

TLDR: "I work at Confluent, the owners of Kafka, and I have determined through my tests that Redpanda's performance is greatly exaggerated." I don't think we can get a less reliable or trustworthy set of performance tests than when someone's paycheck depends on the outcome of those tests. If Redpanda's performance were found to be better, would he really publish the test results?

I don't think who does the benchmark, any benchmark, matters as long as they're open about how it was done, what properties were set, ideally why they were set, and what their results were. The big picture goal is to ostensibly be able to reproduce such benchmarks.

But I've found through industry that most benchmarks, especially for infrastructure software, are performed by the vendors. The burden for standing up the system(s) to pull off the benchmark is usually high enough that independents are rarely going to take up that banner and do it themselves.

Also, notably closed source systems, some vendors don't license their software to allow public benchmarks.

So, transparency is all we can really hope for.

I remember the halcyon days of the database wars with the vendors publishing new benchmarks seemingly ever month. Fun to watch "Lies, damn lies, and statistics" rear up on its hind legs and roar. And some of the monster clusters of hardware these folks put together were legion.

Similarly I enjoyed when Sun was publishing JEE benchmarks on cheap hardware running Glassfish against MySQL. At least they were publishing on these smaller systems more akin to what many companies may run internally in contrast to these million dollar cluster benchmarks BEA and Oracle were publishing.

Finally, just to throw this out, modern hardware is just extraordinary. Hard to appreciate how fast modern machines are if you didn't live with them in the old days.

Were in the glory days where we, most of we, simply don't care. Off the shelf hardware running untuned servers with reasonable algorithms have so much bandwidth and capability, just gets harder and harder to saturate today.

Re: Kafka vs. Redpanda performance – do the claims add up?

#46

Post by Spivak »

agallego wrote: post #22

alex here, original author of redpanda is hard to respond to a 6-part blog series content - released all at once - on an HN thread. - what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. - the kafka partition model of one segment per partition could be optimized in both arch - the benefit for all of us, is that all of thes…

> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here.

Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server.

Re: Kafka vs. Redpanda performance – do the claims add up?

#47

Post by skyde »

Author say "Redpanda incorrectly claim Kafka is unsafe because it doesn’t fsync - it is not true".

If you don't Fsync the batch, it's possible the server would send response to client saying data was written successfully while the batch is still just in memory and then the server loose power and never write it to disk.

Maybe the author have a different definition of unsafe but to me if it's not ACID it's unsafe!

Re: Kafka vs. Redpanda performance – do the claims add up?

#48

Post by deschutes »

globalreset wrote: post #21

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. And then in this article it's explained how Kafka is actually unsafe: > Kafka may h…

You can't eliminate the risk of data loss, only control for it. fsync is one such control. Empirically, having separate power failure domains strongly controls for the power loss risk.

In the tail there are all kinds of things that will lose you data. I've actually seen systems lose data with the fsync every message strategy on simultaneous power loss. There was latent corruption of the filesystem due to a kernel bug. After power cycling a majority of nodes had unrecoverable filesystems.

In my experience, even on modern flash the cost of fsync is non trivial. It pessimizes io. You can try to account for this with group commit / batching but but generally the batch window needs to be large relative to network rtt to be effective.

fsync is much more necessary on single primary systems.

Re: Kafka vs. Redpanda performance – do the claims add up?

#49

Post by skyde »

nemothekid wrote: post #40

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…

Exactly I will never ever try MongoDB because of that. A database that do not fsync should not be called a database.

Re: Kafka vs. Redpanda performance – do the claims add up?

#50

Post by skyde »

skyde wrote: post #47

Author say "Redpanda incorrectly claim Kafka is unsafe because it doesn’t fsync - it is not true". If you don't Fsync the batch, it's possible the server would send response to client saying data was written successfully while the batch is still just in memory and then the server loose power and never write it to disk. Maybe the author have a different definition of unsafe but to me if it's not ACID it's unsafe!

A topic partition can lose some messages without compromising the correctness of the data replication protocol itself.

But I don't think anyone would call a configuration where you ca lose message a safe configuration.

Post reply on HN