21–30 of 39 posts
I recently rewrote some streaming code to slurping. While there are theoretical advantages to streaming, unless you really really need them the drawbacks are not worth it. My comment in the code was "Streaming is hard, memory is cheap."
I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…
The issue is that the structures to handle streaming are not often first class (in languages other than Go, Rust, and JavaScript). And when they are, the the third party interfaces are not. Eventually your data will have to reach the DB and it would bottleneck. Nobody wants to use KV storage all the time, SQL systems and ORMs need to step up their concurrency game.
[1]: https://en.wikipedia.org/wiki/Algorithms_for_calculating_var...
For instance, I'm working on a file-based CRDT-style distributed, journaling database (for single-user but multi-device) and the "diffs" for a single device are stored in a journal. Now in retrospect, it's obvious that I don't need to store the entire journal of diffs in memory (this is a "streaming" architecture), but on my first couple of passes, because it was easier, I stored everything in memory.
After playing around with some toy projects using my database, I realized this wouldn't scale (why load all of history into memory if you don't need it?) and not only that my system lends itself better to a "file stream", i.e. don't store everything in mem, just append to existing files or read from them as needed. Seems obvious now, but when building it out, my first inclination wasn't to do it that way.
Some data formats need writing a length in the header. Or reading requires metadata stored after the data. Some algorithms need two passes, or can be streamed over columns when you're getting rows.
If you've been around long enough, you learned to program in a severely memory-constrained environment: any solution _besides_ a streaming solution was unworkable.
https://en.m.wikipedia.org/wiki/Backpressure_routing
And nodejs's site even has a article for it.
https://nodejs.org/es/docs/guides/backpressuring-in-streams/
However, even the nodejs runtime itself already provide mechanism to let you handle backpressuring easily. Most people still don't do and make their program crash when streaming target is stuck for whatever reason.