Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka vs. Redpanda performance – do the claims add up?

jack-vanlightly.com

Post reply on HN
Print view View original

51–60 of 147 posts

Re: Kafka vs. Redpanda performance – do the claims add up?

#51

Post by insanitybit »

Spivak wrote: post #46
agallego wrote: post #22

alex here, original author of redpanda is hard to respond to a 6-part blog series content - released all at once - on an HN thread. - what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. - the kafka partition model of one segment per partition could be optimized in both arch - the benefit for all of us, is that all of thes…

> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every…

> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server.

Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.

Re: Kafka vs. Redpanda performance – do the claims add up?

#52

Post by dilyevsky »

morelisp wrote: post #34
dilyevsky wrote: post #29

Earlier quoted context omitted.

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw

My gut feeling is that if your only AZ goes down (or all your AZs simultaneously), you're going to lose data period because your producers are now all stuck, your APIs are unavailable, etc. Whether the data loss begins at the exact moment power failed or a couple minutes before doesn't matter, vs. the additional cost to fsync constantly. I mean it's good to know all the failure modes, but at the end of the day it's a…

producers stuck != data loss (if you use transactional commits at least). If you run in multiple regions you dont need multi az in a lot of architectures

Re: Kafka vs. Redpanda performance – do the claims add up?

#53

Post by dilyevsky »

ceencee wrote: post #33
dilyevsky wrote: post #29

Earlier quoted context omitted.

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw

Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t a thing in the cloud at least. There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary. Data loss in kafka isn't a real problem that hurts real world users at all.

> There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary

1. Don't have to do it by message

2. It's used by many distributed db engines, kafka and (i think) zk are the outliers here, not the other way around

Re: Kafka vs. Redpanda performance – do the claims add up?

#54

Post by [deleted] »

skyde wrote: post #49
nemothekid wrote: post #40

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…

Exactly I will never ever try MongoDB because of that. A database that do not fsync should not be called a database.

[deleted]

Re: Kafka vs. Redpanda performance – do the claims add up?

#56

Post by eternalban »

dilyevsky wrote: post #53
ceencee wrote: post #33

Earlier quoted context omitted.

Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t a thing in the cloud at least. There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary. Data loss in kafka isn't a real problem that hurts real world users at all.

> There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary 1. Don't have to do it by message 2. It's used by many distributed db engines, kafka and (i think) zk are the outliers here, not the other way around

Kafka is not a "db engine". zk is a "db engine" in the same way 'DNS' is a "db engine".

Re: Kafka vs. Redpanda performance – do the claims add up?

#57

Post by morelisp »

dilyevsky wrote: post #52
morelisp wrote: post #34

Earlier quoted context omitted.

My gut feeling is that if your only AZ goes down (or all your AZs simultaneously), you're going to lose data period because your producers are now all stuck, your APIs are unavailable, etc. Whether the data loss begins at the exact moment power failed or a couple minutes before doesn't matter, vs. the additional cost to fsync constantly. I mean it's good to know all the failure modes, but at the end of the day it's a…

producers stuck != data loss (if you use transactional commits at least). If you run in multiple regions you dont need multi az in a lot of architectures

I don't mean because of some misfeature in the Kafka protocol, I mean because events are still coming in but have nowhere to go. Unless you built a spill as wide as your Kafka cluster. Which isn't worth it, so no one does it.

Re: Kafka vs. Redpanda performance – do the claims add up?

#58

Post by Spivak »

insanitybit wrote: post #51
Spivak wrote: post #46

Earlier quoted context omitted.

> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every…

> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.

I suppose but that's the trade-off for performance. You have to design your system so that can't happen. Which if you're cloud then it's deploying multi-az, if you're coloing then paying for racks with separate power and/or having battery so you have time to fsync and shut down and if you're fully on-prem then you don't need my advice.

Or I suppose just pay for a managed service from someone who does that for you.

Re: Kafka vs. Redpanda performance – do the claims add up?

#59

Post by haggy »

nemothekid wrote: post #40

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…

Kafka, unlike Mongo DB, relies on recovery/replication instead of fsync:

https://jack-vanlightly.com/blog/2023/4/24/why-apache-kafka-...

Kafka has never tried to hide that fact and it does not, in any way, make Kafka unsafe.

Re: Kafka vs. Redpanda performance – do the claims add up?

#60

Post by llama052 »

We really wanted to try redpanda, but operationally it does not appear to be very k8s* native and infact looks like a lot of one off hand holding to get it working properly.

Hopefully that can get ironed out in the future. Until then we will stick with the Strimzi operator and kafka.

Also Confluent is absolutely pricing themselves out of the market. We looked at their self hosted confluent operator and they wanted something like $9k per node, when they do nothing but provide an operator. Insanity.

Post reply on HN