Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Streaming joins are hard

estuary.dev

Post reply on HN
Print view View original

21–30 of 60 posts

Re: Streaming joins are hard

#21

Post by crazygringo »

BeefWellington wrote: post #9
crazygringo wrote: post #7

Can someone explain what the use case is for streaming joins in the first place? I've written my fair share of joins in SQL. They're indispensable. But I've never come across a situation where I needed to join data from two streams in real time as they're both coming in. I'm not sure I even understand what that's supposed to mean conceptually. It's easy enough to dump streams into a database and query the database bu…

I'll use a contrived example here to explain what the value of streaming the data itself is. Let's say you run a large installation that has a variety of very important gauges and sensors. Due to the size and complexity of this installation, these gauges and sensors need to be fed back to a console somewhere so that an overseer role of sorts can get that big picture view to ensure the installation is functioning full…

I definitely understand the value of streaming. Your gauges example is great.

What I don't understand is streaming joins. None of your gauge values need to join to anything.

And if they did -- if something needed to join ID values to display names, presumably those would sit in a database, not a different stream?

Re: Streaming joins are hard

#22

Post by ryzhyk »

d0mine wrote: post #19
ryzhyk wrote: post #18

The correct way to think about the problem is in terms of evaluating joins (or any other queries) over changing datasets. And for that you need an engine designed for *incremental* processing from the ground up: algorithms, data structures, the storage layer, and of course the underlying theory. If you don't have such an engine, you're doomed to build layer of hacks, and still fail to do it well. We've been building…

Is it related to Differential Dataflow / timely dataflow https://github.com/TimelyDataflow/differential-dataflow

We have our own formal model called DBSP: https://docs.feldera.com/papers

It is indeed inspired by timely/differential, but is not exactly comparable to it. One nice property of DBSP is that the theory is very modular and allows adding new incremental operators with strong correctness guarantees, kind of LEGO brick for incremental computation. For example we have a fully incremental implementation of rolling aggregates (https://www.feldera.com/blog/rolling-aggregates), which I don't think any other system can do today.

Re: Streaming joins are hard

#23

Post by lsuresh »

neeleshs wrote: post #6

"Unlike batch tables, streams are infinite. You can't "just wait" for all the rows to arrive before performing a join." I view batch tables as simply a given state of some set of streams at a point in time. Running the same query against "batch" tables at different points in time yields different results (assuming the table is churning over time).

Your mental model is spot on and described quite well here: https://current.confluent.io/2024-sessions/streaming-queries...

Re: Streaming joins are hard

#24

Post by crazygringo »

GeneralMayhem wrote: post #15
crazygringo wrote: post #7

Can someone explain what the use case is for streaming joins in the first place? I've written my fair share of joins in SQL. They're indispensable. But I've never come across a situation where I needed to join data from two streams in real time as they're both coming in. I'm not sure I even understand what that's supposed to mean conceptually. It's easy enough to dump streams into a database and query the database bu…

Event correlations are a typical one. Think about ad tech: you want every click event to be hydrated with information about the impression or query that led to it. Both of those are high-volume log streams. You want to end up with the results of: ``` select * from clicks left join impressions on (clicks.impression_id=impressions.id) ``` but you want to see incremental results - for instance, because you want to feed…

That's helpful, thanks.

I was definitely under the impression that ad impressions and clicks would be written to databases immediately and queried from there.

I'm still having a hard time imagining in what case you'd need a "live" aggregating display that needed to join data from multiple streams, rather than just accumulating from individual streams, but I guess I can imagine that there are circumstances where that would be desired.

Thanks!

Re: Streaming joins are hard

#25

Post by jrockway »

crazygringo wrote: post #24
GeneralMayhem wrote: post #15

Earlier quoted context omitted.

Event correlations are a typical one. Think about ad tech: you want every click event to be hydrated with information about the impression or query that led to it. Both of those are high-volume log streams. You want to end up with the results of: ``` select * from clicks left join impressions on (clicks.impression_id=impressions.id) ``` but you want to see incremental results - for instance, because you want to feed…

That's helpful, thanks. I was definitely under the impression that ad impressions and clicks would be written to databases immediately and queried from there. I'm still having a hard time imagining in what case you'd need a "live" aggregating display that needed to join data from multiple streams, rather than just accumulating from individual streams, but I guess I can imagine that there are circumstances where that…

I think it can be challenging to get that much data to a single database. For example, you probably don't want to send every "someone moused over this ad" event in Japan to a datacenter in us-east-1. But if you do the aggregation and storage close to the user, you can emit summaries to that central server, backing some web page where you can see your "a 39-year-old white male moused over this ad" count go up in real time.

How important ads are is debatable, but if you're an ad company and this is what your customers want, it's an implementation that you might come up with because of the engineering practicality.

Re: Streaming joins are hard

#26

Post by bob1029 »

> Streaming data isn't static like tables in databases—it's unbounded, constantly updating, and poses significant challenges in managing state.

I don't really see the difference between tables & streams. Data in tables changes over time too. You can model a stream as a table with any degree of fidelity you desire. In fact, I believe this could be considered a common approach for implementing streaming abstractions.

Re: Streaming joins are hard

#27

Post by incr_me »

ryzhyk wrote: post #22
d0mine wrote: post #19

Earlier quoted context omitted.

Is it related to Differential Dataflow / timely dataflow https://github.com/TimelyDataflow/differential-dataflow

We have our own formal model called DBSP: https://docs.feldera.com/papers It is indeed inspired by timely/differential, but is not exactly comparable to it. One nice property of DBSP is that the theory is very modular and allows adding new incremental operators with strong correctness guarantees, kind of LEGO brick for incremental computation. For example we have a fully incremental implementation of rolling aggregat…

Are you aware of any efforts to apply DBSP's theory to a general programming language/environment? From my perspective, DDlog was the most inspiring project in the field of incremental computation, but it seems like all of these projects just lead to implementations of streaming databases or other similar commercial products that fit into Data™ pipelines (no offense). Incremental computation pops up everywhere, from databases to business logic to UI rendering and video game graphics, and I have this hunch that if the problem could be solved at a fundamental level and in an accessible way, we could have revolutionary gains for programmers and programs.

Re: Streaming joins are hard

#28

Post by pgwhalen »

bob1029 wrote: post #26

> Streaming data isn't static like tables in databases—it's unbounded, constantly updating, and poses significant challenges in managing state. I don't really see the difference between tables & streams. Data in tables changes over time too. You can model a stream as a table with any degree of fidelity you desire. In fact, I believe this could be considered a common approach for implementing streaming abstractions.

When one queries a table though, it's only query at one point in time. Querying a stream implies that your result set is a stream as well, which introduces a whole separate set of complexities to worry about both as an implementor of the query engine and a client.

Re: Streaming joins are hard

#29

Post by matlin »

crazygringo wrote: post #7

Can someone explain what the use case is for streaming joins in the first place? I've written my fair share of joins in SQL. They're indispensable. But I've never come across a situation where I needed to join data from two streams in real time as they're both coming in. I'm not sure I even understand what that's supposed to mean conceptually. It's easy enough to dump streams into a database and query the database bu…

We apply incremental, streamable "joins" (relational queries) for real-time syncing between application client and server. I think much of the initial research in this space was around data pipelines but the killer app (no pun intended) is actually in app development

Re: Streaming joins are hard

#30

Post by zbentley »

crazygringo wrote: post #24
GeneralMayhem wrote: post #15

Earlier quoted context omitted.

Event correlations are a typical one. Think about ad tech: you want every click event to be hydrated with information about the impression or query that led to it. Both of those are high-volume log streams. You want to end up with the results of: ``` select * from clicks left join impressions on (clicks.impression_id=impressions.id) ``` but you want to see incremental results - for instance, because you want to feed…

That's helpful, thanks. I was definitely under the impression that ad impressions and clicks would be written to databases immediately and queried from there. I'm still having a hard time imagining in what case you'd need a "live" aggregating display that needed to join data from multiple streams, rather than just accumulating from individual streams, but I guess I can imagine that there are circumstances where that…

Live-updated aggregates are quite common in this area. Consider metered billing ("discontinue this ad after it has been served/clicked/rendered X times"), reactive segmentation ("the owner of a store has decided to offer a discount to anyone that viewed but did not purchase products X, Y, and Z within a 10 minute period"), or intrusion detection ("if the same sequence of routes is accessed quickly in rapid succession across the webserver fleet, regardless of source IP or UA, send an alert").

In a very large number of cases, those streams of data are too large to query effectively (read: cheaply or with low enough latency to satisfy people interested in up-to-date results) at rest. With 100ks or millions of events/second, the "store then query" approach loses fidelity and affordability fast.

Post reply on HN