Would Arroyo be an alternative to Confluent KSQL?
If you look at the page, I guess the answer is yes. Or more precisely, an alternative to Flink written in rust.
Show HN: Arroyo – Write SQL on streaming data
11–20 of 35 posts
Re: Show HN: Arroyo – Write SQL on streaming data
#12This is a really exciting project! I recently learned about https://github.com/vmware/database-stream-processor which builds on a new theoretical foundation and claims to be 9x faster than Flink. It is also written in Rust, and there is a compiler from SQL to Rust executables. Can you comment on the differences?
Re: Show HN: Arroyo – Write SQL on streaming data
#13This is a really exciting project! I recently learned about https://github.com/vmware/database-stream-processor which builds on a new theoretical foundation and claims to be 9x faster than Flink. It is also written in Rust, and there is a compiler from SQL to Rust executables. Can you comment on the differences?
As an aside - we are evaluating different streaming engines to power data projections for Prisma.io. Excited to see support for Debezium coming in v0.4.0.
Would be interesting to somehow make Arroyo run the Nexmark benchmark so we can clearly compare to Flink and DBSP: https://liveandletlearn.net/post/vmware-take-3-experience-wi...
Re: Show HN: Arroyo – Write SQL on streaming data
#14Would love to know how you look at tools list Materialize in comparison
Re: Show HN: Arroyo – Write SQL on streaming data
#15This is a really exciting project! I recently learned about https://github.com/vmware/database-stream-processor which builds on a new theoretical foundation and claims to be 9x faster than Flink. It is also written in Rust, and there is a compiler from SQL to Rust executables. Can you comment on the differences?
As an aside - we are evaluating different streaming engines to power data projections for Prisma.io. Excited to see support for Debezium coming in v0.4.0. Would be interesting to somehow make Arroyo run the Nexmark benchmark so we can clearly compare to Flink and DBSP: https://liveandletlearn.net/post/vmware-take-3-experience-wi...
Hi there! We actually already have a built-in Nexmark source. It's pretty useful for developing new capabilities, and available as a source out of the box.
Just read through the DBSP docs and it looks like it is working in a similar space. The biggest differences in my mind are around distribution and reliability. Arroyo works across a cluster of machines and has built in fault tolerance, while for DBSP that's still just planned for the future.
(I'm the co-creator of Arroyo, for context)
Re: Show HN: Arroyo – Write SQL on streaming data
#16Very exciting, how is feature parity with tinybird?
Re: Show HN: Arroyo – Write SQL on streaming data
#17Between Flink, Spark and KSQL, streaming is very JVM centric. It is nice to see more non JVM projects emerge.
I am not sure about your premise that the operations side is difficult. It tends to be submitting a job to a cluster in Flink or Spark.
The harder barrier to entry is the functional style of transformation code. Even though other frameworks have it, I think the SQL API as the first class citizen is the bigger differentiator.
Re: Show HN: Arroyo – Write SQL on streaming data
#18Very exciting, how is feature parity with tinybird? https://www.tinybird.co/
I am not sure of specifics on features, but I think the fundamental difference is that Arroyo is a stream processing engine i.e., it doesn’t have a database, whereas Tinybird has the statefulness afforded by ClickHouse as its primary data store. Arroyo would be more like Flink, Tinybird would be more like ClickHouse.
Disclaimer: I work for Tinybird.
Re: Show HN: Arroyo – Write SQL on streaming data
#19Earlier quoted context omitted.
As an aside - we are evaluating different streaming engines to power data projections for Prisma.io. Excited to see support for Debezium coming in v0.4.0. Would be interesting to somehow make Arroyo run the Nexmark benchmark so we can clearly compare to Flink and DBSP: https://liveandletlearn.net/post/vmware-take-3-experience-wi...
Hi there! We actually already have a built-in Nexmark source. It's pretty useful for developing new capabilities, and available as a source out of the box. Just read through the DBSP docs and it looks like it is working in a similar space. The biggest differences in my mind are around distribution and reliability. Arroyo works across a cluster of machines and has built in fault tolerance, while for DBSP that's still…
Thank you!
I'll try to get something set up to compare performance of the two on the same machine.
Re: Show HN: Arroyo – Write SQL on streaming data
#20Slightly off-topic.
"Arroyo" is a Spanish word meaning creek, or stream