21–30 of 32 posts
The arrow ecosystem nets you a great compute implementation, storage (parquet), and a great RPC framework (arrow flight).
Arrow has been the most exciting piece of technology I've seen in the last few years. The ecosystem being built around it is amazing, and it's standardizing a bunch of disparate data ecosystems. The arrow ecosystem nets you a great compute implementation, storage (parquet), and a great RPC framework (arrow flight).
How does it compare to DuckDB, which is an Arrow-compatible OLAP SQL database, easy to embed and just plain awesome?
So if you have a stream of sensor data from a bunch of IoT devices, you might use Arroyo to run “live” queries over the flowing data, and you might use DuckDB to do analytical queries over archive data
Although they are both hammers to whack nails with I guess, so a lot of use cases likely work well to solve with both systems
How does it compare with things like: 1. https://github.com/bytewax/bytewax 2. https://github.com/pathwaycom/pathway
I recently read this article (https://materializedview.io/p/from-samza-to-flink-a-decade-o...) about Flink and it commented on Flink grew to fit all of these different use cases (applications, analytics and ETL) with disjoint requirements that Confluent built kafka-streams, ksql and connector for. What of those would you say Arroyo is better suited for?
How does it compare to DuckDB, which is an Arrow-compatible OLAP SQL database, easy to embed and just plain awesome?
This is a different use case - DuckDB is large queries over your entire dataset; Arroyo is continuous queries executing over a live data stream. So if you have a stream of sensor data from a bunch of IoT devices, you might use Arroyo to run “live” queries over the flowing data, and you might use DuckDB to do analytical queries over archive data Although they are both hammers to whack nails with I guess, so a lot of u…
I guess it's still not really designed to work well with infinite relations, but then again neither is SQL really.
Earlier quoted context omitted.
This is a different use case - DuckDB is large queries over your entire dataset; Arroyo is continuous queries executing over a live data stream. So if you have a stream of sensor data from a bunch of IoT devices, you might use Arroyo to run “live” queries over the flowing data, and you might use DuckDB to do analytical queries over archive data Although they are both hammers to whack nails with I guess, so a lot of u…
DuckDB processes data lazily, and can easily be integrated with custom synthetic relations. I guess it's still not really designed to work well with infinite relations, but then again neither is SQL really.
In my experience DuckDB is a batch tool, vs this thing which is a continuous stream tool
Arrow has been the most exciting piece of technology I've seen in the last few years. The ecosystem being built around it is amazing, and it's standardizing a bunch of disparate data ecosystems. The arrow ecosystem nets you a great compute implementation, storage (parquet), and a great RPC framework (arrow flight).
I only really know Arrow from Feather and from Pandas 2.0+. Can you recommend where I could learn more?
Earlier quoted context omitted.
DuckDB processes data lazily, and can easily be integrated with custom synthetic relations. I guess it's still not really designed to work well with infinite relations, but then again neither is SQL really.
Maybe I’ve not used DuckDB enough but: How would you set it up to consume, say, a Kafka stream, apply some transformations or windowed aggregates and continuously output the result? In my experience DuckDB is a batch tool, vs this thing which is a continuous stream tool
Arrow has been the most exciting piece of technology I've seen in the last few years. The ecosystem being built around it is amazing, and it's standardizing a bunch of disparate data ecosystems. The arrow ecosystem nets you a great compute implementation, storage (parquet), and a great RPC framework (arrow flight).
https://arrow.apache.org/docs/format/ADBC.html
And nanoarrow but it seems like the adoption has been pretty a little slow. Perhaps a lot of the projects that I would like to see it in may actually depend on a wider range of arrow functionality.