Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Databases over a Weekend

denormalized.io

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Building Databases over a Weekend

#11

Post by hantusk »

knuckleheads wrote: post #10
Gepsens wrote: post #5

I remember 2 years ago someone proposed adding stream processing in datafusion and PRs followed. But IMO stream processing is an entirely different beast, some people could use the sql engine of df for it though. There are rust projects like Arroyo

I’ve been messing around with sql and stream processing off and on the last few months via https://github.com/zmaril/bpfquery and then https://github.com/zmaril/zquery , so I very much feel this comment. I didn’t want to build out my own stream processing architecture in bpfquery, it was getting pretty tough pretty fast, so I switched over to a datafusion backend in zquery in the hopes that it could do stream process…

I agree. So many disparate solutions. The streaming sql primitives are by themselves good enough (e.g. `tumble`, `hop` or `session` windows), but the infrastructural components are always rough in real life use cases.

crossing fingers for solutions like `https://github.com/feldera/feldera` to be wrapped in a nice database, `https://materialize.com/` to solve their memory issues, or `https://clickhouse.com/docs/en/materialized-view` to solve reliable streaming consumption.

Various streaming processing frameworks often have domain specific languages with a lot of limitations of how to express aggregations and transformations.

Re: Building Databases over a Weekend

#12

Post by knuckleheads »

hantusk wrote: post #11
knuckleheads wrote: post #10

Earlier quoted context omitted.

I’ve been messing around with sql and stream processing off and on the last few months via https://github.com/zmaril/bpfquery and then https://github.com/zmaril/zquery , so I very much feel this comment. I didn’t want to build out my own stream processing architecture in bpfquery, it was getting pretty tough pretty fast, so I switched over to a datafusion backend in zquery in the hopes that it could do stream process…

I agree. So many disparate solutions. The streaming sql primitives are by themselves good enough (e.g. `tumble`, `hop` or `session` windows), but the infrastructural components are always rough in real life use cases. crossing fingers for solutions like ` https://github.com/feldera/feldera ` to be wrapped in a nice database, ` https://materialize.com/ ` to solve their memory issues, or ` https://clickhouse.com/docs/e…

Yeah I have a feeling something like polars for streaming would be super popular and useful, but it just hasn't happened yet. It's much easier to just do say kafka and a long running python script and write out the transformations by hand, than it is to use anything on the market right now. None of the current streaming processors want to be embedded as far as I can tell, that's not where the money is. They all want to be paid to run it in the cloud for you and follow that vc playbook model. Which, fair! I do think there's a lot of space out that isn't being occupied though and I hope somebody tries to fill it soon.

(As an aside, feldera doesn't want to be embedded into your app, materialize either, and clickhouse might just pull a great streaming library out from nowhere, they seem to be good at just doing stuff like that).

Re: Building Databases over a Weekend

#13

Post by def- »

hantusk wrote: post #11
knuckleheads wrote: post #10

Earlier quoted context omitted.

I’ve been messing around with sql and stream processing off and on the last few months via https://github.com/zmaril/bpfquery and then https://github.com/zmaril/zquery , so I very much feel this comment. I didn’t want to build out my own stream processing architecture in bpfquery, it was getting pretty tough pretty fast, so I switched over to a datafusion backend in zquery in the hopes that it could do stream process…

I agree. So many disparate solutions. The streaming sql primitives are by themselves good enough (e.g. `tumble`, `hop` or `session` windows), but the infrastructural components are always rough in real life use cases. crossing fingers for solutions like ` https://github.com/feldera/feldera ` to be wrapped in a nice database, ` https://materialize.com/ ` to solve their memory issues, or ` https://clickhouse.com/docs/e…

> [...] `https://materialize.com/` to solve their memory issues [...]

Disclaimer: I work at Materialize

Recently there have been major improvements in Materialize's memory usage as well as using disk to swap out some data.

I find it pretty easy to hook up to Postgres/MySQL/Kafka instances: https://materialize.com/blog/materialize-emulator/

Re: Building Databases over a Weekend

#14

Post by maximus93 »

Great discussion here! At AI Squared, we have also been exploring the evolving landscape of stream processing and SQL engines. While batch engines like DataFusion excel at handling static data, we recognize the challenges around integrating streaming capabilities and infrastructure seamlessly.

Our focus has been on simplifying data activation pipelines with tools like Multiwoven, which aims to bridge the gap between static and dynamic data needs by supporting connectors for both traditional databases and real-time platforms like Kafka. However, the need for more embedded, developer-friendly streaming solutions is clear, and it’s exciting to see the progress in projects like Arroyo, Materialize, and ClickHouse.

For us, the balance lies in usability and flexibility—how can we empower teams to embed robust data capabilities (whether streaming or batch) into their workflows without overloading on infrastructure complexity? As this ecosystem evolves, we’re optimistic about collaborating and contributing to solutions that make streaming SQL as accessible as traditional SQL.

Looking forward to seeing how this space develops—and kudos to the teams pushing boundaries! https://github.com/Multiwoven/multiwoven/

Post reply on HN