Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streaming joins are hard

estuary.dev

Post reply on HN
Print view View original

11–20 of 60 posts

Re: Streaming joins are hard

#11

Post by hamandcheese »

It seems intuitive to me that a correct streaming join is impossible without an infinite buffer and strong guarantees on how events are ordered. The number of real world systems offering both of those guarantees is zero. Anyone espousing streaming joins as a general solution should be avoided at all costs, particularly if they have a title that contains "architect" or "enterprise" (god forbid both in the same title).

At best, it is a trick to be applied in very specific circumstances.

Re: Streaming joins are hard

#12

Post by mattxxx »

JOINs are just hard period. When you're operating at a large scale, you need to be thinking about exactly how to partition + index your data for the types of queries that you want to write with JOINs.

Streaming joins are so hard, that they're an anti pattern. If you're using external storage to make it work, then your architecture has probably gone really wrong or you're using streams for something that you shouldn't.

Re: Streaming joins are hard

#13

Post by hotstickyballs »

BeefWellington wrote: post #9
crazygringo wrote: post #7

Can someone explain what the use case is for streaming joins in the first place? I've written my fair share of joins in SQL. They're indispensable. But I've never come across a situation where I needed to join data from two streams in real time as they're both coming in. I'm not sure I even understand what that's supposed to mean conceptually. It's easy enough to dump streams into a database and query the database bu…

I'll use a contrived example here to explain what the value of streaming the data itself is. Let's say you run a large installation that has a variety of very important gauges and sensors. Due to the size and complexity of this installation, these gauges and sensors need to be fed back to a console somewhere so that an overseer role of sorts can get that big picture view to ensure the installation is functioning full…

Should’ve just cached the output of group bys.

Re: Streaming joins are hard

#14

Post by tshaddox »

crazygringo wrote: post #7

Can someone explain what the use case is for streaming joins in the first place? I've written my fair share of joins in SQL. They're indispensable. But I've never come across a situation where I needed to join data from two streams in real time as they're both coming in. I'm not sure I even understand what that's supposed to mean conceptually. It's easy enough to dump streams into a database and query the database bu…

Isn't the use case just any time you want a client to essentially subscribe to an SQL query and receive message every time the result of that SQL query changes?

Re: Streaming joins are hard

#15

Post by GeneralMayhem »

crazygringo wrote: post #7

Can someone explain what the use case is for streaming joins in the first place? I've written my fair share of joins in SQL. They're indispensable. But I've never come across a situation where I needed to join data from two streams in real time as they're both coming in. I'm not sure I even understand what that's supposed to mean conceptually. It's easy enough to dump streams into a database and query the database bu…

Event correlations are a typical one. Think about ad tech: you want every click event to be hydrated with information about the impression or query that led to it. Both of those are high-volume log streams.

You want to end up with the results of:

``` select * from clicks left join impressions on (clicks.impression_id=impressions.id) ```

but you want to see incremental results - for instance, because you want to feed the joined rows into a streaming aggregator to keep counts as up to date as possible.

Re: Streaming joins are hard

#16

Post by tombert »

A large part of my job in the last few months has been in the form figuring out how to optimize joins in Kafka Streams.

Kafka Streams, by default, uses either RocksDB or an in-memory system for the join buffer, which is fine but completely devours your RAM, and so I have been writing something more tuned for our work that actually uses Postgres as the state store.

It works, but optimizing JOINs is almost as much of an art as it is a science. Trying to optimize caches and predict stuff so you can minimize the cost of latency ends up being a lot of “guess and check” work, particularly if you want to keep memory usage reasonable.

Re: Streaming joins are hard

#17

Post by ryzhyk »

hamandcheese wrote: post #11

It seems intuitive to me that a correct streaming join is impossible without an infinite buffer and strong guarantees on how events are ordered. The number of real world systems offering both of those guarantees is zero. Anyone espousing streaming joins as a general solution should be avoided at all costs, particularly if they have a title that contains "architect" or "enterprise" (god forbid both in the same title).…

A streaming join indeed requires an unbounded buffer in the most general case when inputs keep growing and any input record on one side of the join can match any record on the other side. However, it does not require inputs to be ordered. An incremental query engine such as Feldera or Materialize can handle out-of-order data and offer strong consistency guarantees (disclaimer: I am a developer of Feldera). In practice, unbounded buffers can often be avoided as well. This may require a specialized join such as as-of join (https://www.feldera.com/blog/asof-join) and some GC machinery.

Re: Streaming joins are hard

#18

Post by ryzhyk »

The correct way to think about the problem is in terms of evaluating joins (or any other queries) over changing datasets. And for that you need an engine designed for *incremental* processing from the ground up: algorithms, data structures, the storage layer, and of course the underlying theory. If you don't have such an engine, you're doomed to build layer of hacks, and still fail to do it well.

We've been building such an engine at Feldera (https://www.feldera.com/), and it can compute joins, aggregates, window queries, and much more fully incrementally. All you have to do is write your queries in SQL, attach your data sources (stream or batch), and watch results get incrementally updated in real-time.

Re: Streaming joins are hard

#19

Post by d0mine »

ryzhyk wrote: post #18

The correct way to think about the problem is in terms of evaluating joins (or any other queries) over changing datasets. And for that you need an engine designed for *incremental* processing from the ground up: algorithms, data structures, the storage layer, and of course the underlying theory. If you don't have such an engine, you're doomed to build layer of hacks, and still fail to do it well. We've been building…

Is it related to Differential Dataflow / timely dataflow https://github.com/TimelyDataflow/differential-dataflow

Re: Streaming joins are hard

#20

Post by ryzhyk »

crazygringo wrote: post #7

Can someone explain what the use case is for streaming joins in the first place? I've written my fair share of joins in SQL. They're indispensable. But I've never come across a situation where I needed to join data from two streams in real time as they're both coming in. I'm not sure I even understand what that's supposed to mean conceptually. It's easy enough to dump streams into a database and query the database bu…

The computational complexity of running an analytical query on a database is, at best, O(N), where N is the size of the database. The computational complexity of evaluating queries incrementally over streaming data with a well-designed query engine is O(delta), where delta is the size of the *new* data. If your use case is well served by a database (i.e., can tolerate the latency), then you're certainly better off relying on the more mature technology. But if you need to do some heavy-weight queries and get fresh results in real-time, no DB I can think of can pull that off (including "real-time" databases).
Post reply on HN