91–100 of 229 posts
How many of the Apache Kafka issues are adressed by switching to Apache Pulsar? I skipped learning Kafka, and jumped right into Pulsar. It works great for our use case. No complaints. But I wonder why so few use it?
Surprised there's no mention of Redpanda here.
How about logging the logs so I can shell into the server to search the messages.
https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.
Not to mention you then also have a KV store. Most problems can be solved with redis + Postgres
Every time another startup falls for the Java + Kafka arguments, it keeps the AWS consultants happier. Fast forward into 2025, there are many performant, efficient and less complex alternatives to Kafka that save you money, instead of burning millions in operational costs "to scale". Unless you are at a hundred million dollar revenue company, choosing Kafka in 2025 is doesn't make sense anymore.
Kafka shouldn't be used for low dataflow systems, true, but you can scale a long way with a simple 3 node cluster.
I'm going to get downvoted for this, but you can literally rebuild Kafka via AI right now in record time using the steps detailed at https://ghuntley.com/z80 . I'm currently building a full workload scheduler/orchestrator. I'm sick of Kubernetes. The world needs better -> https://x.com/GeoffreyHuntley/status/1915677858867105862
Go on then. Post the repo when you have :)
Earlier quoted context omitted.
It would make sense for a highly succesful but stable java project to be replaced like that, but since I'm in the java world, it's usually replaced with another java project. I could provide examples myself, but I'm not convinced it's about java vs c++ or go: hadoop, cassandra, zookeeper
As an outsider, Java looks like a language that can be very fast but it seems like certain idiomatic practices or patterns lead to over-engineered and thus sometimes also slow projects. The Factory > joke comes to mind.
I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.
The worst part of Kafka, for me, is managing the cluster. I don't really like the partitioning and the almost hopelessness that ensues when something goes wrong. Recovery is really tricky. Granted it doesn't happen often, if you plan correctly, but the possibility of going wrong in the partitioning and replication makes updates and upgrades nightmare fuel.
Then we settled into an era where server rooms grew and workloads demanded horizontal scaling and for the high profile users running an odd number of processes was a rounding error and we just stopped doing it.
But we also see this issue re-emerge with dev sandboxes. Running three copies of Kafka, Redis, Consul, Mongo, or god forbid all four, is just a lot for one laptop, and 50% more EC2 instances if you spin it up in the Cloud, one cluster per dev.
I don’t know much Kafka, so I’ll stick with Consul as a mental exercise. If you take something like consul, the voting logic should be pretty well contained. It’s the logic for catching up a restarted node and serving the data that’s the complex part.
Earlier quoted context omitted.
Also remember that NATS was donated to the CNCF a while back, and as a result people built a huge ecosystem around it. Easy to forget.
Relevant: https://www.cncf.io/blog/2025/04/24/protecting-nats-and-the-...
https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.
I greatly prefer redis streams. Not all the same features, but if you just need basic streams, redis has the dead simple implementation I always wanted. Not to mention you then also have a KV store. Most problems can be solved with redis + Postgres