21–30 of 39 posts
> Rather than producing output data and then exiting, like a Hadoop job, a continuous MapReduce program continues to update its results incrementally and in real time as new data is added So keeping track of min/max/average as you add new data is now "continous MapReduce"? Don't get me wrong, a data platform that ingests data and computes useful user-defined aggregates from that sounds useful. But this article feels…
MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.
Isn't the entire field of software just people patting themselves on the back for some extremely basic relational algebra? I don't know what I should be proud of when I learn something new, it all seems extremely basic compared as soon as I learn it.
It’s pretty easy to simplify things down until they sound unimpressive.
> Rather than producing output data and then exiting, like a Hadoop job, a continuous MapReduce program continues to update its results incrementally and in real time as new data is added So keeping track of min/max/average as you add new data is now "continous MapReduce"? Don't get me wrong, a data platform that ingests data and computes useful user-defined aggregates from that sounds useful. But this article feels…
Yeah, the article is an odd spin on what they're building.
> min/max/average as you add new data
Or a 2D kernel density estimate for your dashboards, a real-time view of 3-neighbors in a graph (nodes+edges definition) sized by log1p(request frequency), .... I find it way easier to write a few custom incremental primitives to piece together into that kind of algorithm than to write such an algorithm from scratch.
I'm not crazy about a general-purpose framework/product that tries to allow incremental updates of AllTheThings™ -- my experience thus far suggests that getting it to do what you want (or perform reasonably) on your own data will require enough kludges that you would have been far better off writing the WholeDamnedThing™ yourself.
If they do only support min/max/average and other simple transforms then that's probably not great; they'd be competing directly with something like QuestDB, which is a phenomenal product I'm leaning toward more and more. You don't need millisecond view update times if you can query the whole db in milliseconds.
MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.
Isn't the entire field of software just people patting themselves on the back for some extremely basic relational algebra? I don't know what I should be proud of when I learn something new, it all seems extremely basic compared as soon as I learn it.
MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.
My interpretation is that is why it’s so brilliant. It’s incredibly simple for the end user conceptually but encapsulates optimizing processing across a distributed file system, fault tolerance, shuffling key value pairs, job stage planning, handling intermediates ect. Hadoop a big data framework that reduces the level of competence required to write data pipelines because it was able to hide a massive amount of comp…
I'm curious about the difference between "continuous MapReduce" and I guess a subgraph in a "differential dataflow" (which I have read about but never really used). https://github.com/TimelyDataflow/differential-dataflow
Interesting but how is this as an alternative to Apache Flink's stream processing model?
A big difference is the removal of windowing: Flink lets you aggregate or join events only so long as they arrive in the same temporally-bound window. You're required to have a window, and it's core to the semantics of your workflow. Flow's model doesn't use windows, and allows for long-distance (in time) joins and aggregations. There's no concept of "late" data in Flow: it just keeps on updating the desired aggregat…
This is if you want to use high level API. If you use lower-level ProcessFunction you have even more flexibility.
MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.
Interesting but how is this as an alternative to Apache Flink's stream processing model?