Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

> Are there examples of project that do this? I'd be very much interested in looking into this.

Nope, I don't believe there are. Unfortunately they don't seem like they're interested in exporting their logical plans to substrait, so there's no obvious way forward.

> DataFusion is primarily a batch OLAP system, so we should be able to support hybrid workloads as well. And definitely agree with you re: Polars dev exp. That is something we are aiming for with our forthcoming Python sdk.

Ah, since this is the case, it might also make sense to tap into the datafusion python bindings which recently got a massive overhaul to have a more similar dev ex as polars (though the docs are still quite a bit behind).

I'm looking forward to seeing what the result will be! I know Ibis also is an option, but with my little bit of playing around with it, I've found it's just the lowest common denominator and doesn't provide as nice of an experience as directly using polars (or whatever query engine api is provided).

"streaming data" can generally be thought of as a sequence of events in time. Think measurements from sensors, mouse click events from uses on a webpage, or access logs from a server. These events are often fed into a message bus type system like Apache Kafka or AWS Kinesis.

Stream processing is a programming paradigm for working with these types of timestamped events and a "stream processing engine" like Denormalized seeks to actually execute stream processing compute jobs against streaming data.

The goal of the engine is to abstract away as much of the low level complexities needed to effectively work with this data at scale and provide an ergonomic way for developers to write and operate streaming applications.