71–80 of 147 posts
We really wanted to try redpanda, but operationally it does not appear to be very k8s* native and infact looks like a lot of one off hand holding to get it working properly. Hopefully that can get ironed out in the future. Until then we will stick with the Strimzi operator and kafka. Also Confluent is absolutely pricing themselves out of the market. We looked at their self hosted confluent operator and they wanted so…
> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…
Exactly I will never ever try MongoDB because of that. A database that do not fsync should not be called a database.
PS: I really miss working with mongodb. It's been almost 7 years since I last used it. I'm surprised I don't see it mentioned very often anymore.
Earlier quoted context omitted.
Apache Software Foundation owns Kafka.
Meh. It's obvious Confluent exploited the status of being an Apache open source project in order to say they were open-source. But look at the make up of the PMC of Kafka and it's completely dominated by Confluent employees or former employees. Nothing gets done without Confluent's approval or best interest at heart.
Earlier quoted context omitted.
> what we can deterministically show is data loss on apache kafka with no fsync() [shouldn't be a surprise to anyone] - stay tuned for an update here. Confluent themselves can show this, the part I'm curious about is whether you can show data loss outside of the known documented failure modes. Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every…
> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.
If you need reliable data storage do not use Kafka or similar technologies.
> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. Respect to the Kafka team as Kafka is an incredible piece of software, but the Mong…
Exactly I will never ever try MongoDB because of that. A database that do not fsync should not be called a database.
> Issue #1 is that in Kafka’s server.properties file has the line log.flush.interval.messages=1 which forces Kafka to fsync on each message batch. So all tests, even those where this is not configured in the workload file will get this fsync behavior. I have previously blogged about how Kafka uses recovery instead of fsync for safety. And then in this article it's explained how Kafka is actually unsafe: > Kafka may h…
Yep kafka by default is setup to lose data, many people dont know or dont care it seems…
Just like using HyperLogLog acceptable in many scenarios, using Kafka also acceptable. I am quite baffled how widespread the misuse of technology.
Need reliable data storage? Use a database.
Earlier quoted context omitted.
> Because I, as any can anyone, show data loss by running a cluster without fsync and simultaneously pulling the plug on every server. Woah, yeah that's a serious problem. Data loss under that scenario is nothing to sneeze at.
I am not entirely sure what is the reason to make Kafka transactional. The original goal was to have a message queue that holds statistical data where the data loss cannot significantly alter the outcome of the analytics performed on the (often incomplete) data. Why are in this argument about fsync and such now? Did something change? If you need reliable data storage do not use Kafka or similar technologies.
I suppose that might have been the original goal, but the current tag line includes "data integration" and "mission-critical".
"Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications."
We really wanted to try redpanda, but operationally it does not appear to be very k8s* native and infact looks like a lot of one off hand holding to get it working properly. Hopefully that can get ironed out in the future. Until then we will stick with the Strimzi operator and kafka. Also Confluent is absolutely pricing themselves out of the market. We looked at their self hosted confluent operator and they wanted so…
our real storage is s3 - local disk is for staging/raft layer. how is that not cloud native. if you are referring to cloud native as k8s it is true that our k8s operator was built mostly for our cloud but we released it... the good news is a new interface (same code) w/ more friendly user-defaults is about to get released. you can track it all on github tho.
Worked with many operators in the wild and anything that gives you more control through CRD/automation and less manual pod intervention is a huge win, let's us bake into our already existing pipelines for deployment and releases also. The Confluent($$$$)/Strimzi operators do well on that front. I'm super excited to have competition in this space!
I'll keep an eye out for the new release!
Earlier quoted context omitted.
I vouched for this comment (can we please not, folks?). Sure but many people dont run across AZ bc it costs a ton of money. Fsync alone dont save you but it sure makes it less likely to suffer data loss. > Does a system that has implemented anti-corruption in the storage layer now get to lambast Redpanda, Pulsar, ZooKeeper etc because they didn't implement that? Sure, why not? I think zk doesn’t do fsync too btw
My gut feeling is that if your only AZ goes down (or all your AZs simultaneously), you're going to lose data period because your producers are now all stuck, your APIs are unavailable, etc. Whether the data loss begins at the exact moment power failed or a couple minutes before doesn't matter, vs. the additional cost to fsync constantly. I mean it's good to know all the failure modes, but at the end of the day it's a…
I wonder if there's an embedded equivalent for such systems? Something like fasterlog but more mature?
I've found nats to be very lightweight, and it can bridge (bidirectional) to kafka.
Edit: Oh it also supports websockets to the browser