Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

231–240 of 295 posts

Re: Kafka is dead, long live Kafka

#231

Post by CyanLite4 »

ryanworl wrote: post #11
cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…

Does Azure’s Append Blob support (and AWS’ lack thereof) provide any inherent performance advantages for Azure vs. AWS?

Re: Kafka is dead, long live Kafka

#232

Post by Spiwux »

> how many partitions do I need? Unclear, but better get it right because you can never change it!

That is.. simply wrong. You can change the number of partitions.

Plus I really don't get the "you need an entire team of engineers to operate Kafka" claim you keep repeating. That is simply not true, speaking from experience. It is certainly expensive to run, but doesn't require a lot of engineering hours in our team.

Re: Kafka is dead, long live Kafka

#233

Post by tiew9Vii »

waynesonfire wrote: post #27

company competing with kafka says kafka is dead, is that accurate?

Kafka the Apache open source project is kind of dead/legacy for Confluent.

Confluent re-wrote the backend of Kafka for their SASS product and use the public Kafka wire protocol for public integration https://www.confluent.io/blog/cloud-native-data-streaming-ka... .

I don’t think the Apache Kafka distribution is going away, Confluent will still have paying customers with support contracts wanting features and the protocol will likely still need to evolve but the open source version will likely be a burden on Confluent with their own private implementation so it’ll be interesting to see how things look in a few years.

https://www.confluent.io/blog/cloud-native-data-streaming-ka...

Re: Kafka is dead, long live Kafka

#234

Post by dikei »

jandrewrogers wrote: post #217
EdwardDiego wrote: post #153

Earlier quoted context omitted.

ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…

> ZooKeeper is rock solid. Moving off it is a mistake, IMO. I’m agnostic about Kafka but ZooKeeper is problematic for many use cases based on personal experience and I wouldn’t recommend it. It can be “rock solid” and still not very good. I’ve seen ZK replaced with alternatives at a few different organizations now because it didn’t work well in practice, and what it was replaced with worked much better in every case.…

To be fair, a lot of people use ZK wrong, then complaint about it.

For example, if you use it like a general purpose KV store like Redis, you'll have a bad time.

Another often encountered mistake is people, thinking it doesn't need to store much data, deploy ZK to a server with slow disk/network. Big mistake, as every write to ZK need to be broadcasted and synced to disk, a bottle-neck in disk and network IOPS will kill your ensembles.

Re: Kafka is dead, long live Kafka

#235

Post by lmm »

fulafel wrote: post #228
lmm wrote: post #197

Earlier quoted context omitted.

To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option. What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to…

I think if you dismiss HA setups of SQL dbs as "you won't get around to operating it properly" the same ops culture will also end up getting many less 9's availability than aspired to with Kafka. (But also of course lots of applications are also fine with the availability that you get from fate-sharing with a single db server)

> I think if you dismiss HA setups of SQL dbs as "you won't get around to operating it properly" the same ops culture will also end up getting many less 9's availability than aspired to with Kafka.

Up to a point. IME Kafka is a lot easier to operate in true HA form than SQL dbs, and a lot more commonly operated that way; Kafka has a reputation for being harder to operate than a typical datastore, but that's usually comparing a HA Kafka setup with a single-node SQL db. And I don't know why, but many otherwise high-quality ops teams seem to have a bizzare blind spot around SQL dbs where they'll tolerate a much lower level of resilience/availability than they would for any other part of the stack.

Re: Kafka is dead, long live Kafka

#236

Post by raxxorraxor »

slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I think many that use it might want use something like ActiveMQ instead. I think such middleware can also be interesting for smaller applications. Especially if they manage messages and data streams between two different companies that like to communicate directly but have arcane software components on each side that fit together like fire and water.

Re: Kafka is dead, long live Kafka

#237

Post by Fiahil »

ryanworl wrote: post #11
cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…

> We have a custom metadata database running in our cloud control plane which handles ordering.

This is the secret sauce, right there. Anybody can host a bunch of topics and artefacts on S3, and have them locally-replicated if they want. However, there is no world where you can have that without a synchronisation service that ensure cursors are uniques and properly ordered.

Re: Kafka is dead, long live Kafka

#239

Post by tiew9Vii »

ryanworl wrote: post #11
cduzz wrote: post #6

Couple questions: 1. aren't you going to get murderous S3 API call bills if you're pushing each message directly into S3? How're you buffering / queuing / coalescing messages durably without local storage? 2. what's the problem with "just" running a kafka cluster in each AZ and not replicating data between AZs until it's time to ETL the data to wherever? AZ1 clients push to AZ1 clusters; AZ2 clients push to AZ2 clust…

[WarpStream co-founder and CTO here] 1. Each WarpStream Agent flushes a file to S3 with all the data for every topic-partition it has received requests for in the last ~100ms or so. This means the S3 PUT operations costs scales with the number of Agents you run and the flushing interval, not the number of topic-partitions. We do not acknowledge Produce requests until data has been durably persisted in S3 and our clou…

Flushing every 100ms means you would end up with lots of tiny files (bytes) in s3 unless you have something out of process re-writing them in to larger blobs similar to Delta lakes optomize?

The lots of tiny files would be really inefficient from throughput and api call perspective in blob storage.

With the acks, you have up to 100ms waiting for the buffer to fill, + s3 put request + your metadata request/response. For high throughput that must have very high latency putting back pressure on partitions?

Re: Kafka is dead, long live Kafka

#240

Post by physicles »

EdwardDiego wrote: post #153
mdaniel wrote: post #143

Earlier quoted context omitted.

> I’m not sure how anyone can hate Kafka? My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy The whole ambiguity around whether ZK is really still needed or not https://kafka.apache.org/documentation/#zk > makes keeping two distributed systems alive and healthy, but don't worry you can't move your production cluster…

ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…

Not my experience at all.

We’ve been running a 3 node cluster for several years, and a significant minority of the times I’ve been paged is because ZK got into a bad state that was fixed by a restart (what bad state exactly? Don’t know, don’t care, don’t have two spare weeks to spend figuring it out). Note that we have proper liveness checks on individual instances, so the issue is more complicated than that.

Migrated to 3.3 with KRaft about half a year ago, and we haven’t had a single issue since. It just runs and we resize the disks from time to time.

Post reply on HN