11–20 of 32 posts
Looking forward to NATS support ;)
A NATS and Jetstream connector is in development by an Arroyo user, and hopefully will be merged into master soon!
There is a request: https://github.com/ArroyoSystems/arroyo/issues/162
In any case, I look forward to it.
I find myself needing to mechanically transform and simplify SQL every now and then, and it hardly seems something out of reach of automation, yet somehow I've never been able to find software that simplifies and transforms SQL source-to-source. When I've last looked, I've only found optimisers for SQL execution plans.
Hi! Just reading the docs, this looks really slick. I had a few questions: - When you create tables, are they always connected to a source? How does that work for the cloud version (ie, source = filesystem? would we just use s3, it seems.) - Does arroyo poll an s3 bucket for new files and automatically ingest? - Are you able to do ALTER TABLE? (What if data, or data types, are mismatched?) - Similarly, am I able to c…
For the SQL interface, both sources and sinks are treated as tables. Sources you SELECT FROM, while sinks you INSERT INTO. Right now it is incumbent on the user to correctly specify the types of a source for deserialization. How getting this wrong behaves is a little source-dependent, as some data formats are stricter. Parquet will fail hard at read-time, while JSON will coerce as best as it is able, optionally dropp…
So if I'm understanding, you actually read data directly from (say) S3? It isn't copied from S3 and stored locally (ie, a bunch of local .arrow files.)
(Apologies if I'm ignorant of the underlying tech - I think this is really cool and just trying to wrap my head around what happens from "I upload some data to S3" and "we get query results")
Earlier quoted context omitted.
For the SQL interface, both sources and sinks are treated as tables. Sources you SELECT FROM, while sinks you INSERT INTO. Right now it is incumbent on the user to correctly specify the types of a source for deserialization. How getting this wrong behaves is a little source-dependent, as some data formats are stricter. Parquet will fail hard at read-time, while JSON will coerce as best as it is able, optionally dropp…
Let me take a look - thank you! So if I'm understanding, you actually read data directly from (say) S3? It isn't copied from S3 and stored locally (ie, a bunch of local .arrow files.) (Apologies if I'm ignorant of the underlying tech - I think this is really cool and just trying to wrap my head around what happens from "I upload some data to S3" and "we get query results")
^We call these Filesystem Sources/Sinks to match terminology present in other streaming systems, but I'm not in love with it.
how would I go about calling python code as a step, say if I wanted to explore a grid of parameters and fit models accordingly?
Earlier quoted context omitted.
A NATS and Jetstream connector is in development by an Arroyo user, and hopefully will be merged into master soon!
I don't see any PRs: https://github.com/ArroyoSystems/arroyo/pulls There is a request : https://github.com/ArroyoSystems/arroyo/issues/162 In any case, I look forward to it.
Earlier quoted context omitted.
A NATS and Jetstream connector is in development by an Arroyo user, and hopefully will be merged into master soon!
I don't see any PRs: https://github.com/ArroyoSystems/arroyo/pulls There is a request : https://github.com/ArroyoSystems/arroyo/issues/162 In any case, I look forward to it.
Not exactly on-topic, but does anyone know of SQL-to-SQL optimisers or simplifiers (perhaps DataFusion would be able to do this)? I work with generated query systems and SQL macro systems that make fairly complex queries quite easy to generate, but often times come up with unnecessary joins/subqueries etc. I find myself needing to mechanically transform and simplify SQL every now and then, and it hardly seems somethi…
Not exactly on-topic, but does anyone know of SQL-to-SQL optimisers or simplifiers (perhaps DataFusion would be able to do this)? I work with generated query systems and SQL macro systems that make fairly complex queries quite easy to generate, but often times come up with unnecessary joins/subqueries etc. I find myself needing to mechanically transform and simplify SQL every now and then, and it hardly seems somethi…
SQLGlot brands itself as an sql optimizer also: https://github.com/tobymao/sqlglot?tab=readme-ov-file#sql-op... but I haven't tried that aspect of it personally