21–28 of 28 posts
Side note: one thing I've really noticed lacks formalism in a lot of programming languages and environments is easily streaming data and performing operations on a data "pipe". I normally dislike javascript, but Node.js actually does a pretty cool job of it with their streams API. Really wish every language had something like that built in, with flow control, etc., so you could e.g. pipe data through compression => e…
> Most of these systems are equally expressive and can emulate each other
But a quick stab at how I think it could fall under this taxonomy:
[structured]
If using the DataFrame API
[unstructured]
If using the DStream API
---
[low temporal locality]
If you don't watermark and let the state grow--which I don't have much experience with,
but I think could possibly run into scalability issues, you'd have to write some
imperative logic to cull state--but it's very necessary for low temporal locality.
[high temporal locality]
If using watermarks, it will close the and take the output for you
---
[internally consistent / internally inconsistent]
Not really sure how Spark falls here, I think if you use the DataFrame API, you can have
it be consistent, but a lot of operations aren't supported, so you may end up having to
switch to DStreams and write code imperatively.
And then further, what if one of the streams is processed faster than each the other in
an aggregation? I'm not sure if there's a way to specific business rules around making
things atomic--but I'm sure you could hack it if you dropped low level enough.
I think this is touched upon in the other article:
> When combining multiple streams it's important to synchronize them so that the outputs
> of each reflect the same set of inputs.
That said, I'm really excited by the future of Spark Structured Streaming--I want to write declarative code and say how my data gets from A to B--not what needs to happen to get from A to B.
Side note: one thing I've really noticed lacks formalism in a lot of programming languages and environments is easily streaming data and performing operations on a data "pipe". I normally dislike javascript, but Node.js actually does a pretty cool job of it with their streams API. Really wish every language had something like that built in, with flow control, etc., so you could e.g. pipe data through compression => e…
No mention of Apache Spark? Thinking about where Spark would fit in here, because it has both structured and unstructured APIs--I think it would fall under most of these categories depending how you use, going back to the comment in the article: > Most of these systems are equally expressive and can emulate each other But a quick stab at how I think it could fall under this taxonomy: [structured] If using the DataFra…
It didn't make it into https://scattered-thoughts.net/writing/internal-consistency-... because it has severe limitations for low temporal locality operations:
* As of Spark 2.4, you can use joins only when the query is in Append output mode. Other output modes are not yet supported.
* As of Spark 2.4, you cannot use other non-map-like operations before joins. Here are a few examples of what cannot be used.
* Cannot use streaming aggregations before joins.
* Cannot use mapGroupsWithState and flatMapGroupsWithState in Update mode before joins.
* There are a few DataFrame/Dataset operations that are not supported with streaming DataFrames/Datasets. Some of them are as follows.
* Multiple streaming aggregations (i.e. a chain of aggregations on a streaming DF) are not yet supported on streaming Datasets.
* Limit and take the first N rows are not supported on streaming Datasets.
* Distinct operations on streaming Datasets are not supported.
* Sorting operations are supported on streaming Datasets only after an aggregation and in Complete Output Mode.
* Few types of outer joins on streaming Datasets are not supported.
The rest of spark doesn't really fit the post at all afaict - there is no streaming or incremental update, just batch stuff.
To me the unstructured area is the future of applications development and the structured area is barren and overhyped. (e.g. 'eventually consistent' analytics is so 2021, because it looks like you are doing something quantitative,except it doesn't matter if you get the right answer. It will get you claps from a certain audience but most people lose patience pretty quick when the 'numbers don't add up'.)
IMO the key with eventual consistency is making it known which bits are currently known to be consistent, and which are still in flux. I realize that this is vague, but how that works depends on what it is you're processing and how much time it takes for consistency to emerge. If you're doing any kind of Important Reporting on eventually consistent data, you'd better make sure that you either know you're only includi…
On the low temporal locality side, the only consistent system I've seen so far is differential dataflow (and materialize) which does internally track which results are consistent and gives you the option to wait for consistent results in the output.
Somewhat off-topic, but I feel like the "patronage economy" is really taking off, with Patreon and GitHub sponsors. Content like this is really interesting, and I've appreciated this blog's various write-ups in the past. There's another great blog, with writings supported by patrons, whose name escapes me (I think the domain ends in something about "lime", with the '.me' ccTLD). Whenever I see those domains on HN, I…
I suspect it's also going to be fragile. The next time we hit a recession or, god forbid, another pandemic, sponsorships will probably be the first thing to go.
But it does help fill gaps. There are a lot of things I've wanted to code, study or write about that I think people will get value from but that I could not get any employer to pay for. So I really appreciate being able to work on things just because they're valuable, rather than because there is a way to capture the value and make a profit.
Somewhat off-topic, but I feel like the "patronage economy" is really taking off, with Patreon and GitHub sponsors. Content like this is really interesting, and I've appreciated this blog's various write-ups in the past. There's another great blog, with writings supported by patrons, whose name escapes me (I think the domain ends in something about "lime", with the '.me' ccTLD). Whenever I see those domains on HN, I…
https://fasterthanli.me/articles/aiming-for-correctness-with... ?
I've been following the rust content, it's a readable and useful (to me) blog