If you've been around long enough, you learned to program in a severely memory-constrained environment: any solution _besides_ a streaming solution was unworkable.
Streaming: a skill gap?
11–20 of 39 posts
Re: Streaming: a skill gap?
#12In addition to streams, Buffers confused me for longer than they should have. Unlike streams, which have an indefinite length, buffers are a fix-sized amount of binary data.
Re: Streaming: a skill gap?
#13I love articles like this in hacker news which teaches something specific
Re: Streaming: a skill gap?
#14I recently rewrote some streaming code to slurping. While there are theoretical advantages to streaming, unless you really really need them the drawbacks are not worth it. My comment in the code was "Streaming is hard, memory is cheap."
Re: Streaming: a skill gap?
#15I recently rewrote some streaming code to slurping. While there are theoretical advantages to streaming, unless you really really need them the drawbacks are not worth it. My comment in the code was "Streaming is hard, memory is cheap."
This sounds quite sad.
Re: Streaming: a skill gap?
#16I recently rewrote some streaming code to slurping. While there are theoretical advantages to streaming, unless you really really need them the drawbacks are not worth it. My comment in the code was "Streaming is hard, memory is cheap."
It really depends on the problem space. Mobile phones and browsers only have so much memory available so you have to be mindful of memory consumption there.
Re: Streaming: a skill gap?
#17I've run into plenty of situations where a streaming approach would be faster. The complexity of it always necessitates making a slower conventional version. (wait for all the data to load into memory and the operate on it) the conventional approach is easier to debug and get working. 90% of the time, the gains from streaming aren't worth the added effort. Generally you only really get value for it when you're proces…
> In terms of business value, a cron job running in a high memory vps will more than satisfy and take much less time to develop.
Yeah, but I hate it... I've worked with a team where we had a cron job do some batch processing every night, but for some large customers it started taking ~12-15 hours to complete, and certain important user operations are locked while it's running. The solution? Running once per week starting on the weekend, with a manual trigger for customers who really need the results ASAP. Tiny effort for that easy fix and the team can continue working on new features, all-sized customers are still mostly happy, but Dijkstra would not have liked this...
Re: Streaming: a skill gap?
#18[deleted]
Re: Streaming: a skill gap?
#19Any time streaming data comes up, I want to point people towards some of the existing research and conceptual tooling under the name of "dataflow processing", which is functionally equivalent to stream processing. https://en.wikipedia.org/wiki/Dataflow There's a lot of interesting ideas which we can use to this day, albeit with some changes to work with modern programming languages and data encoding frameworks.
This area of research is fascinating - my favorite part of research in this area last 6 months is finding DAGs everywhere. More or less Dataflow models work by lifting the computational dependencies into data and breaking each compute step into a small enough piece so that they're composable and can replicate across machines arbitrarily. The dependencies between each step is usually modeled as some kind of DAG in a s…
Is DAG in this context "Directed acyclic graph"?
Re: Streaming: a skill gap?
#20Earlier quoted context omitted.
This area of research is fascinating - my favorite part of research in this area last 6 months is finding DAGs everywhere. More or less Dataflow models work by lifting the computational dependencies into data and breaking each compute step into a small enough piece so that they're composable and can replicate across machines arbitrarily. The dependencies between each step is usually modeled as some kind of DAG in a s…
Is DAG in this context "Directed acyclic graph"?
Indeed