How do Pulsar and Kafka compare these days?
Kafka is dead, long live Kafka
71–80 of 295 posts
Re: Kafka is dead, long live Kafka
#72Can this be used without S3?
Re: Kafka is dead, long live Kafka
#73Can this be used without S3?
(WarpStream co-founder)
It requires some form of object storage (S3, GCS, Azure blob storage, etc). You could use minio if you want to host the storage yourself, although I think I need to make a few changes to make that configurable in the agent.
For local development you can use an in-memory or local file storage, but not for "real" usage.
Re: Kafka is dead, long live Kafka
#74Can I use minio?
Re: Kafka is dead, long live Kafka
#75Well, one thing's for sure. Running Kafka on discrete VMs on a cloud provider "by the book" is ludicrously expensive. I remember having a very simple discussion with quite a few customers about both Kafka and Hadoop that boiled down to this: Why replicate data at the VM/disk level when those disks are already provided as a fully redundant system? (in this case it was Azure storage, which provides locally, redundant,…
Or, you know, use ephemeral data storage with your brokers like the good folks intended...
Re: Kafka is dead, long live Kafka
#76Earlier quoted context omitted.
Related to 1. If I understood corrently the agent generates single object per each flushing interval containing all data accross all topics it has received. Does this mean that when reading the consumer needs to read multiple partition data simultaneously to access just single partition? How about scaling consumers horizontally how does WarpStream Agent handle horizontal partitioning of the stream from consuming side…
[WarpStream co-founder here] That is correct about flushing. RE: consuming. The TLDR; is that the agents in an availability zone cluster with each other to form a distributed file cache such that no matter how many consumers you attach to a topic, you will almost never pay for more than 1 GET request per 4MiB of data, per zone. Basically when a consumer fetches a block of data for a single partition, that will trigge…
Is there a reason you built that cache layer yourself (rather than each node "just" running its own sidecar MinIO instance, that write-throughs to the origin object store?)
Re: Kafka is dead, long live Kafka
#77> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…
how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?
Re: Kafka is dead, long live Kafka
#78> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…
I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem.
I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.
Re: Kafka is dead, long live Kafka
#79I’m so happy to see that someone has finally built such thing. I’ve been asking for tiered storage in Kafka since 2016 ( https://gruchalski.com/posts/2016-05-08-the-case-for-kafka-c... ) but this approach is so much better. Kafka itself could solve this with tiered storage OR, at least, allowing adding volumes at runtime. But neither is possible. Very cool, have to find some time to take this for a test drive.
Just fyi, tiered storage is coming at some point in Kafka: https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...