Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

github.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

#21

Post by ambrood »

theLiminator wrote: post #20

Are you going to support OLAP use cases as well? I haven't yet found a really nice hybrid batch/streaming query engine with dataframe support. Ideally, you'd support an api similar to Polars (which I have found to be the nicest thus far). It'd also be important/useful to support Python udfs (think numpy/jax/etc.). It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execu…

DataFusion is primarily a batch OLAP system, so we should be able to support hybrid workloads as well. And definitely agree with you re: Polars dev exp. That is something we are aiming for with our forthcoming Python sdk.

> It'd also be important/useful to support Python udfs (think numpy/jax/etc.).

Yep that's our longterm gameplan.

> It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execute polars logical plans but with a streaming source, that would be huge.

Are there examples of project that do this? I'd be very much interested in looking into this.

Re: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

#22

Post by debadyutirc »

drawnwren wrote: post #12

What differentiates you from i.e. Arroyo and Fluvio?

I work with the creators of Fluvio at InfinyOn.

Fluvio is an edge to core cloud native streaming engine built from the ground up in rust. Compiles to a single 37 Meg binary and deploys on ARM64 devices.

We just released the first public beta version of Stateful DataFlow. Stateful DataFlow is a framework for building unbounded distributed stream processing based on wasm that runs on Fluvio streams.

We are going for a Lean alternative to Kafka + Flink with a user experience of Ruby on Rails.

BTW, Stateful DataFlow has integrations with Arrow, Polars, and the ability to use SQL for dataframes, and other wasm compatible programming languages to express business logic. And Fluvio has Rust, Python, and JS clients.

Re: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

#23

Post by lhnz »

ambrood wrote: post #19
lhnz wrote: post #18

Do you have plans to make the data sources pluggable instead of being Kafka specific?

we absolutely do, the library itself is designed to be extensible. we are currently working on adding webhooks as one of our sources. are there are any specific connectors/sources you'd be interested in?

I have lots of HTTP endpoints that we poll with a cursor but actually the underlying data is very large (we work with snapshots of it) and updates very frequently and eventually we'll move to something else (e.g. interact directly with the underlying services with capnproto) so really it would just be useful to be able to define these sources ourselves. I'm working doing full-stack engineering at an HFT currently and we were thinking of using DataFusion to allow users to join, query and aggregate the data in realtime but I haven't attempted this yet (and to do so means integrating with what currently exists as I don't have time to rewrite all of the services).

Re: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

#25

Post by snthpy »

emgeee wrote: post #9

Other founder here -- we've been working on this now for several months and have had a lot of fun building on top of arrow and datafusion

Hi,

I previously built pq (https://github.com/PRQL/prql-query) as a side project using PRQL, Arrow, DataFusion, and DuckDB in Rust but unfortunately my life got too busy to maintain it.

I've been looking for work in a related area to make it easier to pick up the torch on that again.

I'd love to chat about the space and share experiences. My colleague on PRQL built https://github.com/aljazerzen/connector_arrow which may also be of interest.

You can reach me at at Google email service.

Re: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

#26

Post by j-pb »

I'd be curious to know what your thoughts on differential/timely dataflow are. Superficially it seems that it might be possible to integrate the existing Rust infrastructure from those libraries with DataFusion and Arrow, which could give you quite a few operators for free, and provide your users with the very nice incremental query/streaming-as-view-maintenance model.

Re: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

#27

Post by snthpy »

theLiminator wrote: post #20

Are you going to support OLAP use cases as well? I haven't yet found a really nice hybrid batch/streaming query engine with dataframe support. Ideally, you'd support an api similar to Polars (which I have found to be the nicest thus far). It'd also be important/useful to support Python udfs (think numpy/jax/etc.). It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execu…

Have you looked at Databend? They support Flink CDC (https://docs.databend.com/guides/load-data/load-db/flink-cdc) so should be able to handle hybrid use cases.

I haven't looked at their Python API but they support PRQL which is a pretty nice and ergonomic interface in my (biased) opinion. See https://docs.databend.com/sql/sql-reference/ansi-sql#support...

Re: Show HN: Denormalized – Embeddable Stream Processing in Rust and DataFusion

#28

Post by stereosky »

Congratulations on launching your project! We spoke back in March at a Kafka Summit London social meetup and talked all things Python and Kafka (I work on https://github.com/quixio/quix-streams). Always great to see a new stream processing project tackle a new segment
Post reply on HN