An opinionated map of incremental and streaming systems
scattered-thoughts.net
An opinionated map of incremental and streaming systems
1–10 of 28 posts
Re: An opinionated map of incremental and streaming systems
#2i found this very interesting.
we have a homegrown push-based streaming library in .net based on reactive (it supports indexed joins of "tables") and we're looking for something more infrastructural as we move to the cloud.
Re: An opinionated map of incremental and streaming systems
#3"Last updated 2021-04-18" (so not 2018?)
Re: An opinionated map of incremental and streaming systems
#4Great taxonomy breakdown.
The low temporal locality part of the design space is particularly interesting. It is extremely general but for many data models "low temporal locality" can mean highly variable windows on a per entity basis and not on the entire collection, which is a challenging criteria for the way most systems are designed. Physical world sensor observation data models are a good example of this. Existing solutions for low temporal locality data models scale quite poorly in practice, even on systems nominally designed for them.
At the limit, low temporal locality platforms converge on the same unusual design requirements as symmetric mixed-workload database kernels. The latter is something we really don't (know how to) build, since it has some interesting computer science constraints. It is a missing piece of technology to make this stuff really work well at scale.
Re: An opinionated map of incremental and streaming systems
#5I feel like this map is missing something to tease apart systems that are not interchangeable in practice (UI state management vs database stuff).
Maybe persistent vs in-memory? Like whether the data is typically completely blown away and recomputed on page reload (e.g., ui state, dom trees, scene graphs).
Re: An opinionated map of incremental and streaming systems
#6Would love to hear more options in that bottom left box with differential dataflow et al, because for me that's where all the really interesting work is happening.
Re: An opinionated map of incremental and streaming systems
#7Somewhat off-topic, but I feel like the "patronage economy" is really taking off, with Patreon and GitHub sponsors. Content like this is really interesting, and I've appreciated this blog's various write-ups in the past.
There's another great blog, with writings supported by patrons, whose name escapes me (I think the domain ends in something about "lime", with the '.me' ccTLD).
Whenever I see those domains on HN, I always click, since I have a high expectation the content will be good, given how it was created.
Re: An opinionated map of incremental and streaming systems
#8"Last updated 2021-04-18" (so not 2018?)
Fixed, thanks!
Re: An opinionated map of incremental and streaming systems
#9I like the unstructured/structured boundary as it mirrors the "do you use a library" (some Rx) or "do you use a client" (like Kafka).
I'm writing a programming-language ( http://www.adama-lang.org/ ) which is reactive, and I'm currently looking at how do I optimize (prematurely, of course). I believe a combination of static analysis and a good runtime working together are the future (at least, for me).
For instance, I currently statically optimize "iterate _table where id == x" to "_table.get(x)", and this was a good optimization. I use the same analysis to introduce some limited indices.
The paradox for my use case (board games) is that going further isn't really needed, and the overhead of optimization starts to cost more than brute force table scans. However, I'm excited about this approach beyond board games.
Re: An opinionated map of incremental and streaming systems
#10To me the unstructured area is the future of applications development and the structured area is barren and overhyped.
(e.g. 'eventually consistent' analytics is so 2021, because it looks like you are doing something quantitative,except it doesn't matter if you get the right answer. It will get you claps from a certain audience but most people lose patience pretty quick when the 'numbers don't add up'.)