Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to make any immutable data structure distributed

unison-lang.org

Post reply on HN
Print view View original

41–50 of 54 posts

Re: How to make any immutable data structure distributed

#41

Post by Guvante »

dwohnitmok wrote: post #31
Guvante wrote: post #28

Earlier quoted context omitted.

Sharing a new copy isn't always easy though. That is why lens are a thing. Unfortunately lens are (last I checked) still not quite easy to grow. The point is neither copying the whole structure nor diffs are as easy as mutex + inner mutation.

Sharing the copy is not what's solved by lenses. In a certain sense lenses are just a way to try to do away with the boilerplate of updating nested immutable data structures. It is very easy to write this boilerplate (in the sense that it is straightforward and hard to mess up). It's just mind-numbingly tedious. Although I don't know what you mean by growing lenses. Mutex + inner mutation is no easier than CAS (which…

CAS is a massive pain in the ass for complex changes. Lenses are a way to point to part of the object which can in theory make it less painful as you can redo a change without redoing the work to setup the change necessarily. Growing just refers to growing a codebase using them by adding them is all.

I am not against FP I think the point of "it is good if you can fix performance" is very accurate. I just think it is also important to acknowledge that the paradigm can be more complex in situations where it is supposed to help leading to a mixed bag.

Similar to distributed databases. Your database can now be phenomenally powerful but you can't do a stored proc anymore without losing that power.

It can be a very effective and totally worth it but it isn't a pure win necessarily.

Re: How to make any immutable data structure distributed

#42

Post by lidatong »

zomglings wrote: post #24
lidatong wrote: post #12

I think there were a lot of great ideas presented in these small examples. 1. Explicitly marking non-local _data_ vs. in-memory. A very cool idea -- languages and libraries I've used all seem to try to make this transparent (unsuccessfully, as indicated by the random serialization errors you get) 2. Making the noise around serialization and RPC transparent to the user 3. Bring the code to the data, not the data to th…

Your use of the word transparent confused me a little. It seems you are using it in the sense that details that are being made transparent are being hidden from the user. But if you make something transparent to the user, it could also mean that you are directly exposing that thing to the user. (Right? I could be wrong about this.) Just confused me and thought it might be useful to you to know that. Not trying to be…

No offense taken - re-reading what I wrote, you make a fair point. The wiki page luckysid linked calls this out:

> Confusingly, the term refers to overall invisibility of the component, it does not refer to visibility of component's internals

Indeed, perhaps "invisible" would have been a better word choice.

Re: How to make any immutable data structure distributed

#43

Post by dwohnitmok »

Guvante wrote: post #41
dwohnitmok wrote: post #31

Earlier quoted context omitted.

Sharing the copy is not what's solved by lenses. In a certain sense lenses are just a way to try to do away with the boilerplate of updating nested immutable data structures. It is very easy to write this boilerplate (in the sense that it is straightforward and hard to mess up). It's just mind-numbingly tedious. Although I don't know what you mean by growing lenses. Mutex + inner mutation is no easier than CAS (which…

CAS is a massive pain in the ass for complex changes. Lenses are a way to point to part of the object which can in theory make it less painful as you can redo a change without redoing the work to setup the change necessarily. Growing just refers to growing a codebase using them by adding them is all. I am not against FP I think the point of "it is good if you can fix performance" is very accurate. I just think it is…

> CAS is a massive pain in the ass for complex changes.

Why is that?

Re: How to make any immutable data structure distributed

#44

Post by jmfldn »

cipheredStones wrote: post #7
jmfldn wrote: post #2

I really hope Unison succeeds. It looks like a really interesting take on FP / distributed systems. The fact that Paul Chiusano and Runar Bjarnason are at the helm is exciting, especially to Scala / FP devs like me who know them as the writers of the legendary "red book". Just based on the rare quality of that book, I'm super interested in other FP projects that they're involved in.

Tangential to the thread, but do you have any other favorite Scala resources you would recommend? I've had only glancing exposure to it, and will soon be taking a role where it's the primary language.

I do indeed. The "red book" aka Functional Programming in Scala I wouldn't recommend to a beginner. It teaches FP at a fairly deep level to the extent of implementing the abstractions yourself. Take a look at that later perhaps. Don't get me wrong, it is utterly brilliant but it wouldn't be the place to start.

I would recommend Essential Scala for the basics. Scala with Cats if you're going to be working with the Typelevel stack (a popular set of FP libraries). Lots of other resources but those two alone will take you far.

http://underscore.io/books/essential-scala/

http://underscore.io/books/scala-with-cats/

Re: How to make any immutable data structure distributed

#45

Post by [deleted] »

lidatong wrote: post #13
pchiusano wrote: post #4

One of the authors of the article here, happy to answer any questions about it!

I had a few questions: 1. The posts mention Spark, but were you (the Unison team) also inspired by dataflow / stream processing engines (e.g. Flink, Beam, Timely Dataflow etc.) or any other technologies? 2. Relatedly, are there any plans to add a stream processing API? Or perhaps that would belong in a 3rd-party library? 3. Broadly, where do you think (in your opinion) Unison fits in the ecosystem of tooling? Is Unis…

[deleted]

Re: How to make any immutable data structure distributed

#46

Post by jmfldn »

LandR wrote: post #3
jmfldn wrote: post #2

I really hope Unison succeeds. It looks like a really interesting take on FP / distributed systems. The fact that Paul Chiusano and Runar Bjarnason are at the helm is exciting, especially to Scala / FP devs like me who know them as the writers of the legendary "red book". Just based on the rare quality of that book, I'm super interested in other FP projects that they're involved in.

> writers of the legendary "red book". If you mean the manning book "Functional Programming in Scala", I don't even use Scala, yet I feel I got so much out of that book. It's the best functional programming book I've read. It would recommend the book to anyone that wants to learn FP, regardless if you will be using Scala.

That's the one. It's great for any FP learner for sure. A great example of how to write a book on a deep topic in an engaging and beautifully structured way.

Re: How to make any immutable data structure distributed

#47

Post by lidatong »

pchiusano wrote: post #25
lidatong wrote: post #13

Earlier quoted context omitted.

I had a few questions: 1. The posts mention Spark, but were you (the Unison team) also inspired by dataflow / stream processing engines (e.g. Flink, Beam, Timely Dataflow etc.) or any other technologies? 2. Relatedly, are there any plans to add a stream processing API? Or perhaps that would belong in a 3rd-party library? 3. Broadly, where do you think (in your opinion) Unison fits in the ecosystem of tooling? Is Unis…

So what we were going for with the core API is that it should be expressive enough for all kinds of distributed programming tasks, not just Spark-like or batch computing stuff. We’re planning on doing several of these articles to show different applications. I think some distributed stream processing library would be super cool and there are lots of sources of inspiration for that like you mention. It would be a more…

Thank you both for the answers!

> the thing that would have helped me most in avoiding some of the missteps might have been a richer test interpreter, where I could have "mocked" a network

This is a really intriguing idea. Debugging distributed can be a huge headache, and I can see a lot of value in having something like that.

> We’re planning on doing several of these articles to show different applications.

Looking forward to future articles and definitely will be following this project!

Re: How to make any immutable data structure distributed

#48

Post by runT1ME »

chrisjc wrote: post #39
runT1ME wrote: post #6

I really think this is where pure FP shines. If you look at the architecture of something like Apache Beam, while you describe your computations in a language like Java or Python, you're really using a DSL for creating an immutable DAG that processes chunks of immutable data, persisiting each chunk (being loose with terms here) for durability, 'modifying' it in the immutable sense and then passing it on to the next e…

Intriguing, but I think I lost you a little in the details and really dying to understand your perspective. 1) creating an immutable DAG that processes chunks of immutable data, 2) persisting each chunk (being loose with terms here) for durability, 3) 'modifying' it in the immutable sense 4) and then passing it on to the next edge and so on Isn't #4 the persisting part, not #2? Maybe I'm confused by what you mean by…

Let's say your immutable DAG has four different 'transformations' (think either a map or a fold on the data). Beam or Spark will take a chunk of your data, partition it somehow (so you get parallelism), and do a transform on the data.

Now if you're doing a bunch of computations off of tens or hundreds machines you don't want to fail the whole thing because one hypervisor crashes or there's a JVM segfault or whatnot. So each step is usually saved to disk via checkpoint and then moved along to the next transformation in batches, so if the subsequent transfomation is where it dies, you have a safe spot to restart that particular computation from.

In addition, your (chunk of) data might need to be shipped to an additional machine to be transformed so you're not so much 'updating' data as you are creating new data in a space efficient way of the changes and shuffling those along the various steps.

Re: How to make any immutable data structure distributed

#49

Post by chrisjc »

runT1ME wrote: post #48
chrisjc wrote: post #39

Earlier quoted context omitted.

Intriguing, but I think I lost you a little in the details and really dying to understand your perspective. 1) creating an immutable DAG that processes chunks of immutable data, 2) persisting each chunk (being loose with terms here) for durability, 3) 'modifying' it in the immutable sense 4) and then passing it on to the next edge and so on Isn't #4 the persisting part, not #2? Maybe I'm confused by what you mean by…

Let's say your immutable DAG has four different 'transformations' (think either a map or a fold on the data). Beam or Spark will take a chunk of your data, partition it somehow (so you get parallelism), and do a transform on the data. Now if you're doing a bunch of computations off of tens or hundreds machines you don't want to fail the whole thing because one hypervisor crashes or there's a JVM segfault or whatnot.…

Ok, makes sense now. Thanks for taking the time to explain.

Re: How to make any immutable data structure distributed

#50

Post by Guvante »

dwohnitmok wrote: post #43
Guvante wrote: post #41

Earlier quoted context omitted.

CAS is a massive pain in the ass for complex changes. Lenses are a way to point to part of the object which can in theory make it less painful as you can redo a change without redoing the work to setup the change necessarily. Growing just refers to growing a codebase using them by adding them is all. I am not against FP I think the point of "it is good if you can fix performance" is very accurate. I just think it is…

> CAS is a massive pain in the ass for complex changes. Why is that?

What are you CASing? If the object is too large you will have contention to the point that you might as well be single threaded. If the object is too small you now have to CAS multiple things which is far from trivial.

CAS is a primitive, it isn't complex itself but it can be complex to work with once you are talking non trivial work. Just like locks. A global lock is dumb simple but a real locking system can be as complex as you will let it.

Post reply on HN