Streaming: a skill gap?
charemza.name
Streaming: a skill gap?
1–10 of 39 posts
Re: Streaming: a skill gap?
#2I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort.
Generally you only really get value for it when you're processing a huge data set continuously or modifying data as its being sent to the user over a websocket for a fairly lengthy bit of time.
In terms of business value, a cron job running in a high memory vps will more than satisfy and take much less time to develop.
Re: Streaming: a skill gap?
#3I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…
> 90% of the time, the gains from streaming aren't worth the added effort.
I gotta disagree with that estimate. Virtually any time I have a backend service operating on (mostly) arbitrarily-sized user input, I use streaming so that I can make better guarantees about how much memory my service needs. This, in turn, lets you give your customers much higher service limits (unless you want to scale your fleet's memory just to handle 100th-percentile style use cases).
The number of times I've seen backend services fall over, with a heap graph that looks like a repeated sawtooth pattern to OOM, because a customer's objects were unusually sized (but within limits..)..
Re: Streaming: a skill gap?
#4I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…
> The complexity of it always necessitates making a slower conventional version.
I agree with this, but I feel that in most cases it's not nessecary complexity. It comes from poor APIs that don't make streaming easy, or mismatch between push-oriented ("pass me each new chunk as it arrives") and pull-oriented ("give me a queue/file/iterator that will yield chunks").
Re: Streaming: a skill gap?
#5Any time streaming data comes up, I want to point people towards some of the existing research and conceptual tooling under the name of "dataflow processing", which is functionally equivalent to stream processing.
https://en.wikipedia.org/wiki/Dataflow
There's a lot of interesting ideas which we can use to this day, albeit with some changes to work with modern programming languages and data encoding frameworks.
Re: Streaming: a skill gap?
#6I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…
Processing time becomes a problem as well when the depth of the call tree across process boundaries starts to climb.
Each service in turn has to request, receive, parse, process, and emit the data. Bandwidth and CPU time turn into latency. Those can start to add up.
Assuming you can stream, doing so in this particular scenario will also improve the latency, not just throughput.
Re: Streaming: a skill gap?
#7I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…
> 90% of the time, the gains from streaming aren't worth the added effort. I gotta disagree with that estimate. Virtually any time I have a backend service operating on (mostly) arbitrarily-sized user input, I use streaming so that I can make better guarantees about how much memory my service needs. This, in turn, lets you give your customers much higher service limits (unless you want to scale your fleet's memory ju…
Yeah this is an important accidental DOS vector, and streaming APIs are a classic way to fix them.
But you do have to be careful that you're not just overloading some other system (like consuming disk space with files that don't need to be retained). Keep good stats on all of your exhaustible resources, kids.
Re: Streaming: a skill gap?
#8I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…
"90% of the time, the gains from streaming aren't worth the added effort."
I... won't go so far as to say "I think", but "I have a pet theory" that part of the reason for this is actually effect rather than cause. That is, developers generally do not think in streaming, so they build libraries that are based on doing things non-streaming, which have libraries built on top of them that assume non-streaming, which have frameworks built on top of them that assume non-streaming, etc. etc. and so on, and the end result is that it's just way harder to get streaming working than it would be if more developers were comfortable with it.
The web world even more so, which for pretty much its entire run has been conceptualized by developers as returning chunks of content, even though the tech nominally had more streaming support than that, being (until recently) TCP sockets under the hood. Web developers even made it a virtue that once a chunk was emitted, all context was dropped on the floor. (I see this as less a virtue than an accidental way old CGI stuff worked that got raised into a requirement.)
Historically speaking, only the minimal things that needed to support streaming to work at all supported it. I am seeing a slow trend towards more streaming-thinking though, and it's getting easier to stream things.
This is an explanation of why I think the quoted text is true, not a disagreement. I think in a more perfect world it wouldn't be true, and I have hope that it won't be true in the medium-term future, but today it often is, depending on details of your local environment.
Re: Streaming: a skill gap?
#9The issue is that the structures to handle streaming are not often first class (in languages other than Go, Rust, and JavaScript). And when they are, the the third party interfaces are not. Eventually your data will have to reach the DB and it would bottleneck. Nobody wants to use KV storage all the time, SQL systems and ORMs need to step up their concurrency game.
Re: Streaming: a skill gap?
#10Any time streaming data comes up, I want to point people towards some of the existing research and conceptual tooling under the name of "dataflow processing", which is functionally equivalent to stream processing. https://en.wikipedia.org/wiki/Dataflow There's a lot of interesting ideas which we can use to this day, albeit with some changes to work with modern programming languages and data encoding frameworks.
This area of research is fascinating - my favorite part of research in this area last 6 months is finding DAGs everywhere. More or less Dataflow models work by lifting the computational dependencies into data and breaking each compute step into a small enough piece so that they're composable and can replicate across machines arbitrarily. The dependencies between each step is usually modeled as some kind of DAG in a simple case the output of one step feeds into another. In more complex cases you have logical (if this then that) dependencies for error handling and branching cases.
I think the Dataflow paper: https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
and the MillWheel paper in particular are good reads about the problem: https://static.googleusercontent.com/media/research.google.c...
What's especially interesting to me is that the same approach is used for rule/workflow engines. Although the use-cases are somewhat different and the graph structure isn't usually the same - they're still modeling compute steps in a graph just with more conditional logic.