Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka vs. Redpanda performance – do the claims add up?

jack-vanlightly.com

Post reply on HN
Print view View original

91–100 of 147 posts

Re: Kafka vs. Redpanda performance – do the claims add up?

#91

Post by threeseed »

skyde wrote: post #49
nemothekid wrote: post #40

> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…

Exactly I will never ever try MongoDB because of that. A database that do not fsync should not be called a database.

MongoDB has been doing fsync by default for over a decade now .

And those that actually had tried it were aware that every client enabled fsync out of the box. So in fact the entire situation was seriously overblown.

But sure let irrational ideology affect your technology decisions. That will work out well.

Re: Kafka vs. Redpanda performance – do the claims add up?

#92

Post by winrid »

llama052 wrote: post #60

We really wanted to try redpanda, but operationally it does not appear to be very k8s* native and infact looks like a lot of one off hand holding to get it working properly. Hopefully that can get ironed out in the future. Until then we will stick with the Strimzi operator and kafka. Also Confluent is absolutely pricing themselves out of the market. We looked at their self hosted confluent operator and they wanted so…

MongoDB Enterprise is 8-20k per node and they just provide management software and support.

Re: Kafka vs. Redpanda performance – do the claims add up?

#93

Post by agallego »

llama052 wrote: post #78
agallego wrote: post #70

Earlier quoted context omitted.

our real storage is s3 - local disk is for staging/raft layer. how is that not cloud native. if you are referring to cloud native as k8s it is true that our k8s operator was built mostly for our cloud but we released it... the good news is a new interface (same code) w/ more friendly user-defaults is about to get released. you can track it all on github tho.

Oh I see the operator now, looks decent. Before the deployment documentation I had found was very manual and full of a lot of pod exec commands. Worked with many operators in the wild and anything that gives you more control through CRD/automation and less manual pod intervention is a huge win, let's us bake into our already existing pipelines for deployment and releases also. The Confluent($$$$)/Strimzi operators do…

totally. we built a new team focused on the dev experience of k8s alone. 90seconds to prod (on a working eks cluster) with TLS, external certs, etc. That's the benchmark we're trying to hit :)

Re: Kafka vs. Redpanda performance – do the claims add up?

#94

Post by winrid »

AtNightWeCode wrote: post #55
globalreset wrote: post #9

EDIT: Thank you for clarification. It is a fair 3 node vs 3 node benchmark. Does this benchmark compare both 3 node Kafka against 3 node Redpanda cluster? It's unclear.

Surely nobody runs 3 Kafka nodes in production, no?

I know of several production clusters at 50m+ ARR companies with three nodes.

Re: Kafka vs. Redpanda performance – do the claims add up?

#95

Post by threeseed »

AtlasBarfed wrote: post #88
uberduper wrote: post #72

Earlier quoted context omitted.

MongoDB moved on from mmap at version ~3.6. WiredTiger can be configured to fsync every commit. Enjoy trying MongoDB! PS: I really miss working with mongodb. It's been almost 7 years since I last used it. I'm surprised I don't see it mentioned very often anymore.

Last I heard of MongoDB it was getting utterly buried by the Jepsen guy, and for anyone that follows distributed systems at some technical level, that is damning. He finds stuff wrong with everything, but that one was particularly damning. MongoDB has always seemed to place write consistency secondary to other priorities (mostly sales / read / features) which is frankly a crap way to do a database, much less a distri…

a) Every distributed database has had serious issues with Jepsen.

b) MongoDB has been growing revenue ~40% year on year for the last few years.

c) PostgreSQL is only a serious competitor for MongoDB if you have small datasets. After all these years PostgresSQL still is ridiculously poor when it comes to clustering, replication etc. Everyone's solution of "just buy a bigger instance" is just laughable.

Re: Kafka vs. Redpanda performance – do the claims add up?

#96

Post by jvanlightly »

sitkack wrote: post #66

I'd like to see a baseline of fio and iperf3 for these same instances so we know how much raw performance is available for disk, network alone and together. Cloud instances have their own performance pathologies, esp in the use of remote disks. As for RP and Kafka performance, I'd love to see a parameter sweep over both configuration dimensions as well as workload. I know this is a large space, but it needs to be don…

These instances can manage up disk throughput up to 2 GB/s (400K IOPS) and network throughout of 25gbps or ~3.1 GB/s.

There are so many dimensions, with configurations, CPU architecture, hardware resources plus all the workloads and the client configs. It gets kind of crazy. I like to use a dimension testing approach where I fix everything but vary one or possibly two dimensions at a time and plot the relationships to performance.

Re: Kafka vs. Redpanda performance – do the claims add up?

#97

Post by datadeft »

ako wrote: post #83
datadeft wrote: post #74

Earlier quoted context omitted.

I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.

Do you have a source/link for that original goal? I wasn’t aware of this, and as such expect that I can rely on kafka for my events. Also, if this is really the case it should be mentioned on the homepage of Kafka. Just checked kafka’s homepage, it mentions mission critical, durable, fault tolerant, stores data safely, zero message loss, trusted… Seems they’ve moved on from their original goal.

"The original use case for Kafka was to be able to rebuild a user activity tracking pipeline as a set of real-time publish-subscribe feeds. This means site activity (page views, searches, or other actions users may take) is published to central topics with one topic per activity type."

https://kafka.apache.org/

Re: Kafka vs. Redpanda performance – do the claims add up?

#98

Post by datadeft »

tyingq wrote: post #77
datadeft wrote: post #74

Earlier quoted context omitted.

I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.

>The original goal was to have a message queue that holds statistical data I suppose that might have been the original goal, but the current tag line includes "data integration" and "mission-critical". "Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications."

I guess you can add any feature to anything. I think this whole investor driven development is just sad.

Re: Kafka vs. Redpanda performance – do the claims add up?

#99

Post by datadeft »

skrtskrt wrote: post #89
datadeft wrote: post #74

Earlier quoted context omitted.

I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.

Kafka is used widely as a persistent event store, and its development features reflect that. Why would I not just turn on fsync or deploy in a distributed pattern for reliability so I can just continue using it instead of ripping it out, benchmarking something new, teaching the entire org something new, potentially negotiating a new contract, and then executing a huge migration?

Just like heroin is widely used a recreational drug. We live in a free world and you can use Kafka as a persistent reliable store, even use it transactionally.

Instead of reading the marketing claims I like to read what @aphyr has to say about data storage systems.

https://aphyr.com/posts/293-call-me-maybe-kafka

Re: Kafka vs. Redpanda performance – do the claims add up?

#100

Post by Dylan1312 »

The biggest point of contention here seems to be over whether kafka can still be considered durable/safe when fsync is disabled.

Seems like it'd be valuable to have a trusted third party like https://jepsen.io/ test it out! (not related, just a fan of their work)

Post reply on HN