Ask HN: Stream processing engine that joins to historical data in Snowflake?

Hey, I'm looking for advice on stream processing engines that allow me to do the following:

1. Required: Write a query that joins an event stream with a historical table in Snowflake 2. Required: Executes in near-real time What are the best options out there? It seems like Apache Flink enables this, but there also seem to be a number of other projects out there that may enable some or all of what I'm describing, including:

- kSQL - Arroyo - Proton - Kafka Streams - Snowflake's Snowpipe Streaming - Benthos - RisingWave - Spark Streaming - Apache Beam - Timely Dataflow and derivatives (Materialize, Bytewax, etc.)

Any recommendations on the best tool for the job? Are there interesting alternatives that I haven't named?

That’s not how stream processing works? It’s absurdly costly to query snowflake for every record that streams in. So then you try to cache the snowflake dataset. But then the cache becomes stale. Honestly this sounds like an XY problem — the problem is stated as X but you’re really trying to solve something else Y.

What is the business requirement? What are the technical specifications required to meet the need? From there, we can start to consider architecture solutions.

Storage is relatively cheap compared to compute. Most stream processors require or at least highly encourage you to provide them direct access to the input data instead of calling an external system.