221–229 of 229 posts
Earlier quoted context omitted.
I'm wondering how much of that is bad developer UX and defaults, and how much of that is inherent complexity in the problem space. Like the article outlines, partitions are not that useful for most people. Instead of removing them, how about having them behind a feature flag, i.e. not on by default. That would ease 99% of users problems. The next point in the article which to me resonates is the lack of proper schema…
> why is there no in-memory kafka server that I can use for simple testing purposes https://github.com/embeddedkafka/embedded-kafka It's for scala. I'm trying to do something similar in Java but haven't got time yet.
This is a question we asked 6 years ago. What if we wrote it in Rust. And leveraged and WASM. We have been at it for the past 6 years. https://github.com/infinyon/fluvio For the past 2 years we have also been building Flink using Rust and WASM. https://github.com/infinyon/stateful-dataflow-examples/
Fluvio looks awesome! Any chance you’re going to be reviving support for the Kafka wire protocol? https://github.com/infinyon/fluvio/issues/4259
This is a question we asked 6 years ago. What if we wrote it in Rust. And leveraged and WASM. We have been at it for the past 6 years. https://github.com/infinyon/fluvio For the past 2 years we have also been building Flink using Rust and WASM. https://github.com/infinyon/stateful-dataflow-examples/
Interesting! How would you say your project compares to Arroyo?
Arroyo is SQL first stream processing. Fluvio is streaming transport which can send data to Arroyo and there is an integration.
Stateful DataFlow and Arroyo are similar in the stream processing pattern and the use of Apache Arrow.
The interfaces are different. Fluvio and Stateful DataFlow support for SQL is the same dialect as columnar SQL supported by Polars. The Fluvio and Stateful DataFlow paradigm is more intricate more expressive and the platform is broader and deeper.
I agree on the head of the line blocking problem and that not everyone needs the per partition ordering. For that I have started to use SQS FIFO with the message grouping key being the logical key for the event/resource. This gives me ordering within the key and not extra ordering across keys. So I don’t have the head of line blocking problem. If I need multiple independent consumers, I just instead publish to SNS FI…
I agree on the head of the line blocking problem and that not everyone needs the per partition ordering. For that I have started to use SQS FIFO with the message grouping key being the logical key for the event/resource. This gives me ordering within the key and not extra ordering across keys. So I don’t have the head of line blocking problem. If I need multiple independent consumers, I just instead publish to SNS FI…
How do you handle DLQ for ordered keys? I assume if you drop a message then you lose semantics
https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.
NATS is also in the process of a open source license rugpull... https://news.ycombinator.com/item?id=43783452