Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka vs. Redpanda performance – do the claims add up?

jack-vanlightly.com

Post reply on HN
Print view View original

31–40 of 147 posts

Re: Kafka vs. Redpanda performance – do the claims add up?

#31

Post by igetspam »

victor106 wrote: post #5
tapoxi wrote: post #3

I'd like to see these on OpenJDK 11, since that's what Confluent is running on and the author makes a point of switching to 17 even though he works for Confluent. In either case, Confluent Platform is ridiculously expensive and approached the costs (licensing alone) for our entire cloud spend. I'd love to see more run-on-k8s alternatives to CFK.

I agree with Confluent pricing, we had the same experience. We switched to pub/sub and Azure Event hubs. I don't even understand why Confluent should price their offering so high. ITs not like Real time is an exclusive service that other platforms don't have.

At my last org, we spent hundreds on confluent and then they did a pricing adjustment and our bill went up 4x. No exaggeration. We moves from kinesis to confluent because it was cheaper. After that, we moved back to save money.

Re: Kafka vs. Redpanda performance – do the claims add up?

#32

Post by igetspam »

pram wrote: post #7
victor106 wrote: post #5

Earlier quoted context omitted.

I agree with Confluent pricing, we had the same experience. We switched to pub/sub and Azure Event hubs. I don't even understand why Confluent should price their offering so high. ITs not like Real time is an exclusive service that other platforms don't have.

Because they have stock they need to pump. I've found talking to Confluent about anything is a complete waste of time unless it's a very specific technical issue. They're always pushing their cloud as the solution, and it's very aggressive.

Really? We had the opposite experience. We got the impression that sales loves to sell against their cloud. Probably commission related.

Re: Kafka vs. Redpanda performance – do the claims add up?

#33

Post by ceencee »

dilyevsky wrote: post #29
jvanlightly wrote: post #27

Earlier quoted context omitted.

Well, that just isn't accurate really. Kafka would need simulteanous VM failure across all AZs. That just doesn't happen in the real world often enough to worry about. It has never happened in Confluent Cloud. RP have a similar issue. Single AZ deployments with local NVMe drives. AZ loses power, a majority of brokers could lose all their data. Then there's data corruption. Fsyncs alone don't save you. The next step w…

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw

Who is running single az deployments who also cares about data loss and availability? Seriously? I’ve personally supported 1000s of kafka deploys and this isn’t a thing in the cloud at least. There is no call for wanting fsync per message, it is an anti pattern and isn’t done because it isn’t necessary. Data loss in kafka isn't a real problem that hurts real world users at all.

Re: Kafka vs. Redpanda performance – do the claims add up?

#34

Post by morelisp »

dilyevsky wrote: post #29
jvanlightly wrote: post #27

Earlier quoted context omitted.

Well, that just isn't accurate really. Kafka would need simulteanous VM failure across all AZs. That just doesn't happen in the real world often enough to worry about. It has never happened in Confluent Cloud. RP have a similar issue. Single AZ deployments with local NVMe drives. AZ loses power, a majority of brokers could lose all their data. Then there's data corruption. Fsyncs alone don't save you. The next step w…

I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw

My gut feeling is that if your only AZ goes down (or all your AZs simultaneously), you're going to lose data period because your producers are now all stuck, your APIs are unavailable, etc. Whether the data loss begins at the exact moment power failed or a couple minutes before doesn't matter, vs. the additional cost to fsync constantly.

I mean it's good to know all the failure modes, but at the end of the day it's also good to know how much handling them will cost, and it's often not worth it.

Re: Kafka vs. Redpanda performance – do the claims add up?

#35

Post by morelisp »

gagejustins wrote: post #2

"I hope you come away with a new appreciation that trade-offs exist, there is no free lunch despite the implementation language or algorithms used. Optimizations exist, but you can’t optimize for everything. In distributed systems you won’t find companies or projects that state that they optimized for CAP in the CAP theorem. Equally, we can’t optimize for high throughput, low latency, low cost, high availability and…

> In distributed systems you won’t find companies or projects that state that they optimized for CAP in the CAP theorem.

This is absolutely rich from the company that keeps promising "exactly once delivery" (with reams of fine print about what "exactly" and "once" and "delivery" mean).

Re: Kafka vs. Redpanda performance – do the claims add up?

#36

Post by agallego »

morelisp wrote: post #30
notfromhere wrote: post #25

Earlier quoted context omitted.

Disabling fsync is dubious. I do find it interesting that Confluent feels the need to respond to RP given the disparities in size, install base, etc.

I've been watching Redpanda for a couple years primarily because I'm interested in their wasm data transformations. In the past 3 months I've heard it mentioned several dozen times by other teams in our company, vs. maybe 2-3 times in the >1y prior. So something seems in the air, and presumably Confluent has noticed. I'm not sure why, Kafka per se doesn't seem to have really dropped any significant balls lately (and…

We’re about to released a revamped wasm and new sdk with prev lessons learned. Should be cool

Re: Kafka vs. Redpanda performance – do the claims add up?

#37

Post by agallego »

Lucasoato wrote: post #14
tapoxi wrote: post #3

I'd like to see these on OpenJDK 11, since that's what Confluent is running on and the author makes a point of switching to 17 even though he works for Confluent. In either case, Confluent Platform is ridiculously expensive and approached the costs (licensing alone) for our entire cloud spend. I'd love to see more run-on-k8s alternatives to CFK.

There's really no reason for Confluent to be so expensive, the pricing depends on so many factors that it's so easy to fuck up and receive stellar bills. Also the fact that they are releasing so many components with restrictive licenses, or that they postponed the Kafka tiered storage feature (that allows you to unload some of the topics data to S3 instead of expensive SSD disks) so that they could squeeze more money…

For long term storage I agree too. The reason we invented our byoc was so that (1) you own your storage and (2) we only charge you for value add

Re: Kafka vs. Redpanda performance – do the claims add up?

#39

Post by agallego »

chalcolithic wrote: post #17

I wonder if there's an embedded equivalent for such systems? Something like fasterlog but more mature?

When you add compaction, indexing, recovery, tiered storage, etc. some things become harder to reason about wrt systems resources if you are embedded.

Re: Kafka vs. Redpanda performance – do the claims add up?

#40

Post by nemothekid »

>Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety.

Respect to the Kafka team as Kafka is an incredible piece of software, but the Mongo guys got torched for eternity for pulling the same shenanigans.

Post reply on HN