111–120 of 201 posts
Earlier quoted context omitted.
Re 2, have you ever tried using Kafka with a non-JVM client?
Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)
Outside of librdkafka and jvm client, it’s gloves off.
Earlier quoted context omitted.
> Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. BookKeeper is a feature though. Allows to scale the partition beyond the capacity of a storage unit. Effectively unlimited retention for a partition. The problem with Kafka is that the broker is tied to storage.
It can be a feature, yep, although Pulsar's tiered storage (offload old data into S3 and then retrieve it if required) is far more of a feature IMO. And BK isn't a feature without cost, it's documentation is... somewhat sparse, and there is significant complexity to maintaining it.
I remember asking for it 5 years ago: https://radek-gruchalski.medium.com/the-case-for-kafka-cold-.... Confluent turned it into a paid feature.
IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.
It seems weird to me that cloud providers do not offer distributed coordination primitives "as a service." I understand there are KV stores but not with watches, locks, etc. in the way that etcd and ZK have them.
1. Locking service
2. Id generator
3. Bloom filter etc.,
Earlier quoted context omitted.
Is there any message ordering guarantee in NATS? With Kafka you can achieve this by using keyed message and messages in the same partition will always be ordered Disclosure: I work for Confluent
I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.
Please separate storage from brokers next
Why though? It's worth noting that Twitter built their own system (EventBus) that Apache Pulsar largely mimics in design (and the people who started Pulsar at Yahoo had worked on EventBus prior), with brokers decoupled from storage, and then eventually just decided to get rid of it and use Kafka. https://blog.twitter.com/engineering/en_us/topics/insights/2... > One catch to this is that for extremely bandwidth-heavy…
The benefit is not just independent scaling of resources, but more useful features like archiving and reading from object storage with infinite history, and faster and more reliable data replication across regions.
IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.
I really love a lot of the software under the Hadoop umbrella, but so much of it assumes a static deployment on bare metal hosts, it's a struggle to use it in "modern" setups (HBase, for example; I miss my old friend).
Earlier quoted context omitted.
What do you use instead?
Cheap managed cloud services, like AWS SQS and Azure Storage Queue (I usually want some kind of persistence for my queues).
Earlier quoted context omitted.
NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.
Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) N…
Not true. Both Nats streaming and the upcoming jetstream (core nats) do.
Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20
Does NATS support (or plan to support) log compaction? Without that, it's very hard to replace Kafka.
Earlier quoted context omitted.
Hijacking seems like a strong word since Jay Kreps is the CEO.
Fair enough, not hijacking. Just presenting the software as open source then requiring you to pay substantially for features that it’d be irresponsible to use the software in production without (e.g. geo replication via mirror maker). Add to that their insistence in claiming “exactly once delivery semantics” from Kafka despite that being provably impossible and I don’t see any reason to trust them as a company or pay…
Exactly once delivery is impossible. Exactly once processing is possible. TBH, semantically there's very little difference between those two from an end user perspective.