The analogy of a library vs a bookstore to explain the distinction between push and pull query engines is quite effective. It really simplifies a complex topic and make it more relatable.
Query Engines: Push vs. Pull
11–13 of 13 posts
Re: Query Engines: Push vs. Pull
#12Earlier quoted context omitted.
This is an interesting point. In Arroyo [0] (the streaming engine I work on) we also use a simple TCP-based backpressure mechanism: we have NxM TCP connections between each operator subtask, behind small in-memory queues. This essentially is relying on the kernel's flow control and backpressure mechanisms instead of building a custom network stack like Flink does. This has worked well in practice, but I think does le…
Interesting - I looked into your code a bit. I found your window aggregation library [1]. You may be interested in looking into the Rust implementation of some of the research work I've been a part of [2]. In Flink, I believe the reason they need to implement their own backpressure system is that they multiplex TCP connections. That is, they have multiple logical streams flowing through a single TCP connection. If th…
Thanks for that link! We've been heavily inspired by the work out of TU Berlin on Scotty (https://tu-berlin-dima.github.io/scotty-window-processor/) but exciting to see more work on this problem. Our efficient window handling is a big area of improvement over what Flink is able to do right now.
For Yep, Flink has to do this themselves because they multiplex. My sense is that when Flink was originally designed in ~2012 Linux didn't support large numbers of TCP connections well (and memory was more of an issue), so they built a network stack that multiplexes many logical streams on top of a single TCP connection.
My (not terribly informed) opinion is that TCP behaviors and semantics are not necessarily optimal for these systems and you get benefits from allowing your systems' scheduler to control prioritization rather than handing it off the kernel.
There are also a new class of datacenter-oriented protocols like Homa (https://homa-transport.atlassian.net/wiki/spaces/HOMA/overvi...) that I think are pretty interesting to solve this class of problems.
Re: Query Engines: Push vs. Pull
#13Earlier quoted context omitted.
This is an interesting point. In Arroyo [0] (the streaming engine I work on) we also use a simple TCP-based backpressure mechanism: we have NxM TCP connections between each operator subtask, behind small in-memory queues. This essentially is relying on the kernel's flow control and backpressure mechanisms instead of building a custom network stack like Flink does. This has worked well in practice, but I think does le…
Interesting - I looked into your code a bit. I found your window aggregation library [1]. You may be interested in looking into the Rust implementation of some of the research work I've been a part of [2]. In Flink, I believe the reason they need to implement their own backpressure system is that they multiplex TCP connections. That is, they have multiple logical streams flowing through a single TCP connection. If th…
Yes, multiplexing data flows that need flow control on top of protocols that have flow control is a difficult problem.
SSHv1 had multiplexing but no flow control in the SSHv1 protocol so if you did a bulk transfer over a forwarded port your interactive session channel could "freeze" if the bulk transfer got flow controlled on the receiving end or if it took up all available bandwidth.
SSHv2 fixed this by adding flow control at the SSHv2 channel layer protocol, but the price paid for this was that that flow control essentially acts as a brake on each channel's bandwidth, so you just can't use SSHv2 for high-bandwidth bulk transfers unless you have implementations tuned for that (meaning that they set huge channel windows).
Nesting flow control just doesn't work very well.