https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.
Also remember that NATS was donated to the CNCF a while back, and as a result people built a huge ecosystem around it. Easy to forget.
What If We Could Rebuild Kafka from Scratch?
61–70 of 229 posts
Re: What If We Could Rebuild Kafka from Scratch?
#62Object storage for Kafka? Wouldn't this 10x the latency and cost? I feel like Kafka is a victim of it's own success, it's excellent for what it was designed, but since the design is simple and elegant, people have been using it for all sorts of things for which it was not designed. And well, of course it's not perfect for these use cases.
> the design is simple and elegant
Kafka is simple and elegant?
Re: What If We Could Rebuild Kafka from Scratch?
#63Earlier quoted context omitted.
Good choice, leaves space to rewrite in rust later, right?
The fact that this is so common I think yearns for a language that is basically python and rust smashed together where a project would have some code in the python side and some code in the rust side intermixed fluidly like how you can drop to asm in C. Don't even really try to make the two halves of the language similar. An embedded interpreter and JIT in rust basically but jostled around a bit to make it more cohes…
Maybe something will eventually crystalize out of mojo?
Re: What If We Could Rebuild Kafka from Scratch?
#64Earlier quoted context omitted.
Writing directly to the datastore ignores the need for queuing the writes. How do you solve for that need?
Why do you need to queue the writes?
some writes might fail, you may need to retry, the data store may be temporarily available etc.
There may be many things that go wrong and how you handle this depends on your data guarantees and consistency requirements.
If you're not queuing what are you doing when a write fails, throwing away the data?
Re: What If We Could Rebuild Kafka from Scratch?
#65Earlier quoted context omitted.
Yeah... It took 4 years to properly integrate Kafka into our pipelines. Everything, like everything is complicated with it: cluster management, numerous semi-tested configurations, etc. My final conclusion with it is that the project just doesn't really know what it wants to be. Instead it tries to provide everything for everybody, and ends up being an unbelievably complicated mess. You know, there are systems that k…
systemd knows very well what it wants to be, they just don't tell anyone. it's real goal is to make Linux administration as useless as windows so RH can sell certifications. tell me the output of systemctl is not as awful as opening the windows service panel.
There are 2 service panels in Windows since 8 and they are quite different...
Re: What If We Could Rebuild Kafka from Scratch?
#66> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a tru…
Alternatively, your write doesn't have to be fire-and-forget: downstream datastores can also write to kafka (this time fire-and-forget) and the initial client can wait for that event to acknowledge the initial write
Re: What If We Could Rebuild Kafka from Scratch?
#67I can't count the number of bad message queues and buses I've seen in my career.
While it would be useful to just blame Kafka for being bad technology it seems many other people get it wrong, too.
Re: What If We Could Rebuild Kafka from Scratch?
#68Object storage for Kafka? Wouldn't this 10x the latency and cost? I feel like Kafka is a victim of it's own success, it's excellent for what it was designed, but since the design is simple and elegant, people have been using it for all sorts of things for which it was not designed. And well, of course it's not perfect for these use cases.
It can increase latency (which can be somewhat mitigated though by having a write buffer e.g. on EBS volumes), but it substantially _reduces_ cost: all cross-AZ traffic (which is $$$) is handled by the object storage layer, where it doesn't get charged. This architecture has been tremendously popular recently, championed by Warpstream and also available by Confluent (Freight clusters), AutoMQ, BufStream, etc. The KIP mentioned in the post aims at bringing this back into the upstream open-source Kafka project.
Re: What If We Could Rebuild Kafka from Scratch?
#69"Faced with such a marked defensive negative attitude on the part of a biased culture, men who have knowledge of technical objects and appreciate their significance try to justify their judgment by giving to the technical object the only status that today has any stability apart from that granted to aesthetic objects, the status of something sacred. This, of course, gives rise to an intemperate technicism that is not…
For each of the items discussed I explicitly mention why they would be desirable to have. How is this engineering for the sake of engineering?
Re: What If We Could Rebuild Kafka from Scratch?
#70> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a tru…
The problem is that you don't know who's listening. You don't want all possible interested parties to hammer the database. Hence the events in between. Arguably, I'd not use Kafka to store actual data, just to notify in-flight.
For read only queries, hammer away I can scale reads nigh infinitely horizontally. There's no secret sauce that makes it so that only kafka can do this.