11–20 of 33 posts
This looks super interesting. I built https://github.com/finos/perspective in a past life but have been out of the streaming analytics game for some time. Nice to see single machine efficiency be a focus, will give this a try and post feedback on github.
this looks so clutch! curious if this was purpose built for the finance industry?
What differentiates you from i.e. Arroyo and Fluvio?
What differentiates you from i.e. Arroyo and Fluvio?
First the similarities: both Arroyo and Denormalized use DataFusion and Arrow and are focused on high-scale, low-latency stateful stream processing.
Arroyo has been around a lot longer and is overall more mature. It's distributed (I believe Denormalized at this point is a single-node engine), supports consistent snapshotting of its state, event time and watermarks, and has a wide range of supported connectors (https://doc.arroyo.dev/connectors). It ships with a control plane, distributed schedulers, and web ui.
But the use cases we're targeting are different. Arroyo programmed via SQL, and is used primarily for real-time data pipelines; we aim to replace Flink SQL and kSQL.
Denormalized (as I understand it) is focused more on data science use cases where it makes sense to have an embedded engine, rather than a distributed one. It's programmed with a Rust dataframe API (and soon Python).
Do you have plans to make the data sources pluggable instead of being Kafka specific?
Ideally, you'd support an api similar to Polars (which I have found to be the nicest thus far).
It'd also be important/useful to support Python udfs (think numpy/jax/etc.).
It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execute polars logical plans but with a streaming source, that would be huge.