Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a streaming SQL engine with Arrow and DataFusion

arroyo.dev

Post reply on HN
Print view View original

1–10 of 32 posts

Re: Building a streaming SQL engine with Arrow and DataFusion

#3

Post by benrutter »

Especially factoring in the streaming capabilities an arrow based SQL database is an exciting prospect!

My assumption is that throughput could be increased quite a bit for loading data into arrow based libaries like polars or pandas since data doesn't have to be converted. Any idea if that works out?

Re: Building a streaming SQL engine with Arrow and DataFusion

#4

Post by necubi »

rahulrs wrote: post #2

SQL streaming engines really seem to be having a moment. As someone who is less familiar with all the players in the space, how should I think about Arroyo vs. streaming databases like Materialize or caching tools like Readyset?

> SQL streaming engines really seem to be having a moment.

I definitely agree! In the past few years, a bunch of folks (including myself) who had been working with Flink/Spark Streaming/KSQL/etc. at large companies decided that the time was right for a new generation of streaming systems and started companies to do that. For myself, seeing how much users struggled to build pipelines on Flink at Lyft inspired me to build Arroyo.

I think it's really exciting after ~5 years of relative stagnation.

> As someone who is less familiar with all the players in the space, how should I think about Arroyo vs. streaming databases like Materialize or caching tools like Readyset?

There are no hard lines (and internally all of these systems look fairly similar) but the products and use cases are pretty different.

To give my gloss:

* Readyset is a very clever cache for your OLTP database that lets you push it into more analytical territory with reasonable performance, but still focused mostly on product use cases; the stream processing system is internal and not exposed to users

* Materialize is designed to provide OLAP materialized views on top of your OLTP database by reading postgres/mysql changefeeds. It gives you up-to-date results for analytical queries without needing to replicate your postgres to snowflake and repeatedly query it.

* Arroyo is a modern Flink, designed for more traditional stream processing use cases. This includes real-time analytics, but is more focused on operational and product use cases like alerting, real-time ML, automated remediation, and streaming ETL.

Also, Arroyo is the only one of these that is fully open source (apache 2) and designed for self hosting.

Re: Building a streaming SQL engine with Arrow and DataFusion

#5

Post by necubi »

benrutter wrote: post #3

Especially factoring in the streaming capabilities an arrow based SQL database is an exciting prospect! My assumption is that throughput could be increased quite a bit for loading data into arrow based libaries like polars or pandas since data doesn't have to be converted. Any idea if that works out?

That's a future direction we're very excited about, particularly being able to run pyarrow-based UDFs on Arroyo state without any serialization overhead.

Re: Building a streaming SQL engine with Arrow and DataFusion

#8

Post by memset »

Hi! Just reading the docs, this looks really slick. I had a few questions:

- When you create tables, are they always connected to a source? How does that work for the cloud version (ie, source = filesystem? would we just use s3, it seems.) - Does arroyo poll an s3 bucket for new files and automatically ingest? - Are you able to do ALTER TABLE? (What if data, or data types, are mismatched?) - Similarly, am I able to change the primary key (ie, clickhouse's ORDER BY or projections?) or change indexes?

Any plans for HTTP as a source? (This is what we build and I'd be happy to prototype an integration!)

Re: Building a streaming SQL engine with Arrow and DataFusion

#9

Post by jnewhouse »

memset wrote: post #8

Hi! Just reading the docs, this looks really slick. I had a few questions: - When you create tables, are they always connected to a source? How does that work for the cloud version (ie, source = filesystem? would we just use s3, it seems.) - Does arroyo poll an s3 bucket for new files and automatically ingest? - Are you able to do ALTER TABLE? (What if data, or data types, are mismatched?) - Similarly, am I able to c…

For the SQL interface, both sources and sinks are treated as tables. Sources you SELECT FROM, while sinks you INSERT INTO. Right now it is incumbent on the user to correctly specify the types of a source for deserialization. How getting this wrong behaves is a little source-dependent, as some data formats are stricter. Parquet will fail hard at read-time, while JSON will coerce as best as it is able, optionally dropping the data instead of failing the job depending on the bad_data parameter: https://doc.arroyo.dev/connectors/overview#bad-data.

Currently we don't support much in the way of changing configuration in external systems, instead focusing on defining long-running pipelines.

What did you have in mind for an HTTP source? We have a polling HTTP source, as well as a WebSocket source:

https://doc.arroyo.dev/connectors/polling-http https://doc.arroyo.dev/connectors/websocket

Post reply on HN