Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to make any immutable data structure distributed

unison-lang.org

Post reply on HN
Print view View original

11–20 of 54 posts

Re: How to make any immutable data structure distributed

#11

Post by Scarbutt »

runT1ME wrote: post #6

I really think this is where pure FP shines. If you look at the architecture of something like Apache Beam, while you describe your computations in a language like Java or Python, you're really using a DSL for creating an immutable DAG that processes chunks of immutable data, persisiting each chunk (being loose with terms here) for durability, 'modifying' it in the immutable sense and then passing it on to the next e…

Besides performance, what other cons exists for immutable data structures in a single standalone system?

Re: How to make any immutable data structure distributed

#12

Post by lidatong »

I think there were a lot of great ideas presented in these small examples.

1. Explicitly marking non-local _data_ vs. in-memory. A very cool idea -- languages and libraries I've used all seem to try to make this transparent (unsuccessfully, as indicated by the random serialization errors you get)

2. Making the noise around serialization and RPC transparent to the user

3. Bring the code to the data, not the data to the code

4. Immutability, FP, and I like the syntax -- no coincidence in its similarities to Haskell and Scala, I'm sure :)

These are all pain points in a lot of existing "big data systems" I've used, where they either solve the problem half-baked or don't address at all. I'm excited to see where this project goes!

Re: How to make any immutable data structure distributed

#13

Post by lidatong »

pchiusano wrote: post #4

One of the authors of the article here, happy to answer any questions about it!

I had a few questions:

1. The posts mention Spark, but were you (the Unison team) also inspired by dataflow / stream processing engines (e.g. Flink, Beam, Timely Dataflow etc.) or any other technologies?

2. Relatedly, are there any plans to add a stream processing API? Or perhaps that would belong in a 3rd-party library?

3. Broadly, where do you think (in your opinion) Unison fits in the ecosystem of tooling? Is Unison specialized at solving big data problems that require distributed execution e.g. something I pull out of my toolbox for a problem I'd otherwise solve with Spark? Or is it a truly general-purpose programming language?

4. With the Tree example, you show the "wrong" way and then the "right" way -- one problem I've experienced in the past is how do you as a new user know the right way. For example, I wrap my entire program in IO then .unsafeRunSync it -- technically following the rules, but got nothing out of doing that. Have you thought about how the APIs could guide the user to the idiomatic solution? Perhaps via types?

By the way, I really loved the FP in Scala book, it was my first step into FP -- thank you and Runar for writing it

Re: How to make any immutable data structure distributed

#14

Post by Guvante »

Scarbutt wrote: post #11
runT1ME wrote: post #6

I really think this is where pure FP shines. If you look at the architecture of something like Apache Beam, while you describe your computations in a language like Java or Python, you're really using a DSL for creating an immutable DAG that processes chunks of immutable data, persisiting each chunk (being loose with terms here) for durability, 'modifying' it in the immutable sense and then passing it on to the next e…

Besides performance, what other cons exists for immutable data structures in a single standalone system?

Sharing immutable data is hard. Mutable throw you throw a lock on it and mutate in place. How do you share mutations when you can't change anything?

To be clear it is certainly a solvable problem (as this post shows) but it can make things difficult.

Re: How to make any immutable data structure distributed

#15

Post by jltsiren »

I dislike the term "immutable data structure", because it has two widely used conflicting meanings:

1. Read-only data structures, which are used by everyone in every language all the time.

2. Persistent data structures, which make the underlying immutable structures kind of mutable by creating new versions and reusing parts from the old versions of the structure.

Re: How to make any immutable data structure distributed

#16

Post by rlmark »

Touche wrote: post #10
pchiusano wrote: post #4

One of the authors of the article here, happy to answer any questions about it!

How do you perform side effects in Unison (like reading a file as an example)?

Hi there, the other article author chiming in here! You can perform side effects in Unison via abilities (our name for algebraic effects) - they're described here: https://www.unisonweb.org/docs/abilities/

Some basic IO functionality is supported by the `base` library (the standard lib) which contains functions like `openFile` which are effectful functions expressed in terms of the `IO` ability.

Re: How to make any immutable data structure distributed

#17

Post by romero-jk »

Guvante wrote: post #14
Scarbutt wrote: post #11

Earlier quoted context omitted.

Besides performance, what other cons exists for immutable data structures in a single standalone system?

Sharing immutable data is hard. Mutable throw you throw a lock on it and mutate in place. How do you share mutations when you can't change anything? To be clear it is certainly a solvable problem (as this post shows) but it can make things difficult.

This is how clojure does it: https://clojure.org/reference/atoms

It's very costly though.

Re: How to make any immutable data structure distributed

#18

Post by nyanpasu64 »

Guvante wrote: post #14
Scarbutt wrote: post #11

Earlier quoted context omitted.

Besides performance, what other cons exists for immutable data structures in a single standalone system?

Sharing immutable data is hard. Mutable throw you throw a lock on it and mutate in place. How do you share mutations when you can't change anything? To be clear it is certainly a solvable problem (as this post shows) but it can make things difficult.

Sharing mutable data without a mutex (which suffers from unbounded contention) is hard. Approaches that work include updating persistent data structures and sharing the new copy, or sharing diff objects over a lock-free queue.

Re: How to make any immutable data structure distributed

#19

Post by mopierotti »

jltsiren wrote: post #15

I dislike the term "immutable data structure", because it has two widely used conflicting meanings: 1. Read-only data structures, which are used by everyone in every language all the time. 2. Persistent data structures, which make the underlying immutable structures kind of mutable by creating new versions and reusing parts from the old versions of the structure.

Aren't those the same things? i.e. 2. are just read-only data structures with functions to create new read-only data structures based on them. (e.g. immutable linked lists that you can filter, concat, etc. with)

Or are you talking about something more specific, like mutable data structures that provide immutable snapshots of the backing data?

Re: How to make any immutable data structure distributed

#20

Post by skyde »

jltsiren wrote: post #15

I dislike the term "immutable data structure", because it has two widely used conflicting meanings: 1. Read-only data structures, which are used by everyone in every language all the time. 2. Persistent data structures, which make the underlying immutable structures kind of mutable by creating new versions and reusing parts from the old versions of the structure.

lol the word "persistent" is itself ambiguous an have conflicting meaning:

1-https://en.wikipedia.org/wiki/Persistence_(computer_science)

In computer science, persistence refers to the characteristic of state of a system that outlives (persists more than) the process that created it.

2- https://en.wikipedia.org/wiki/Persistent_data_structure

In computing, a persistent data structure or not ephemeral data structure is a data structure that always preserves the previous version of itself when it is modified.

Post reply on HN