Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Managing mutable data in Elixir with Rust

lambdafunctions.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Managing mutable data in Elixir with Rust

#31

Post by clarkema »

lvass wrote: post #28
clarkema wrote: post #27

Earlier quoted context omitted.

That's an interesting point that I should perhaps have covered in the original article. The real code that this is based on is in fact hidden behind a GenServer for this exact reason -- to maintain the expectations of other Elixir code that has to interact with it. The advantage of the escape hatch, as another commenter mentions, is allowing efficient sparse mutations of a large chunk of data, without having to pay a…

Have you measured performance? If mutating from Elixir like this can bring serious benefits, maybe there's a place for mutable versions of libraries like Explorer and Nx.

Explorer does actually use Rust (and polars) for a lot of its work -- its one on the libraries I looked at while figuring out my memory management issues.

Re: Managing mutable data in Elixir with Rust

#32

Post by lvass »

clarkema wrote: post #31
lvass wrote: post #28

Earlier quoted context omitted.

Have you measured performance? If mutating from Elixir like this can bring serious benefits, maybe there's a place for mutable versions of libraries like Explorer and Nx.

Explorer does actually use Rust (and polars) for a lot of its work -- its one on the libraries I looked at while figuring out my memory management issues.

But would it benefit from mutating the value of one reference? At the moment it does not do that, right?

Re: Managing mutable data in Elixir with Rust

#34

Post by dpflan »

parthdesai wrote: post #12
WolfeReader wrote: post #11

Earlier quoted context omitted.

I'm not sure Joe Armstrong would agree with your comment.

After a certain scale, it actually does start to slow you down https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...

This is a great engineering blogpost. Other good ones from Discord on scaling (elixir, rust, cassandraDB vs. scyllaDB:

- https://discord.com/blog/how-discord-scaled-elixir-to-5-000-... — continually improving elixir

- https://discord.com/blog/how-discord-stores-trillions-of-mes... — moving to scyllaDB

Re: Managing mutable data in Elixir with Rust

#36

Post by rkangel »

clarkema wrote: post #27
mgdev wrote: post #25

Rustler is great. Though this gets me thinking about how you can maintain as many Elixir invariants and conventions as possible, even while escaping them under the covers. Being able to call FeGraph.set/2 and have db actually be mutated violates Elixir's common call patterns, even if it's technically allowed. For example: I wonder if it wouldn't be more "erlangy"/"elixiry" to model the mutable ops behind a genserver…

That's an interesting point that I should perhaps have covered in the original article. The real code that this is based on is in fact hidden behind a GenServer for this exact reason -- to maintain the expectations of other Elixir code that has to interact with it. The advantage of the escape hatch, as another commenter mentions, is allowing efficient sparse mutations of a large chunk of data, without having to pay a…

Did you consider a port (written in Rust) instead of a NIF?

When you're presenting a GenServer like message passing interface a port is a natural fit, with none of the risks related to linking a NIF into the VM itself.

(admittedly those risks are much lower with Rust than C)

Re: Managing mutable data in Elixir with Rust

#37

Post by unoti »

galaxyLogic wrote: post #29
unoti wrote: post #17

Earlier quoted context omitted.

> Immutable data is not a “foundation of scalability and robustness”. It may not be the only way to get to scalability and robustness. But it certainly is the cornerstone of how Erlang gets there. 1. First, the way Erlang treats data ensures that every piece of data can be sent over the wire by default. This helps pave the way for another amazing characteristic of Erlang, and that is when you refer to and use an obje…

> you instantiate it you communicate with it by sending it messages Does this mean a web-endpoint has to be immutable? If you send it the same parameters multiple times, is it required to respond with the same response every time? If not, does that not mean it is in fact mutable? I read elsewhere that in Elixir programs, there is no difference in messaging a local "agent" or a remote one? The caller does not know whe…

When they say all the data in Erlang/Elixir is immutable, it does not mean that there is no state. There's definitely data in the programs that change values, because like you point out, how in the heck can you write a useful system that doesn't have any state anywhere?

State or data that changes values is typically put in one of 3 different places:

1. On the stack. It's pretty typical to have a function like handle_message(app_state, request). The current state of the app is in the call parameters. At the end of this message, handle_call() would call itself again with the new updated state. Somewhere else in the system we keep track of the last value of that state, and if handle_call crashes, we just use the last state.

2. Another place to hold state is in external storage somewhere.

3. The third main place to hold state is via references to other objects, which do #1 or #2 above.

Regarding whether there is a difference between messaging local versus remote objects-- there's an operator for sending messages to another object. It works the same for local and remote. I think it's possible to inspect the actor address and see where it is, but the mechanism works the same.

Re: Managing mutable data in Elixir with Rust

#38

Post by prophesi »

wredue wrote: post #22
dimitrios1 wrote: post #16

Earlier quoted context omitted.

There isn't some universe where there exists a list of axiomatic, unfalisifable proofs for what or what doesn't constitute the foundations of scalability and robustness, rather, you have a tradition of development practices that have much more often lead you to scalability and robustness than than the alternatives, and Joe Armstrong was someone who trailblazed that tradition in blood sweat and tears so to speak, as w…

The author claimed that immutability is a foundation for scalability and robustness. I reject that claim. In this comment, you simultaneously agree and disagree with me. I don’t give a shit what Joe Armstrong says about immutability because the facts are the facts: 1) immutability cause performance problems 2) immutability significantly limits how you can manage data, which is counter to what computers are meant to d…

I think the main thing (in the context of Elixir) is that immutability gives you concurrency out-of-the-box, which in turn gives you predictability and scalability. It is very easy to spin up a couple of nodes that can communicate with each other thanks to this ecosystem.

Re: Managing mutable data in Elixir with Rust

#39

Post by clarkema »

lvass wrote: post #32
clarkema wrote: post #31

Earlier quoted context omitted.

Explorer does actually use Rust (and polars) for a lot of its work -- its one on the libraries I looked at while figuring out my memory management issues.

But would it benefit from mutating the value of one reference? At the moment it does not do that, right?

No, it doesn't -- looking at the website that's an explicit trade-off of pure performance vs 'Elixir-ish-ness'. It would certainly break a lot of expectations to have data mutating like that without it being hidden away somewhere, so I can understand why they went that way.

In my case the data I'm dealing with is more of a store than a single data item, so I'm leaning on the example of things like ETS. Also it's within a single application rather than being a large generally-available library, so the trade-offs are different. It would be interesting to know if they did tests though.

Re: Managing mutable data in Elixir with Rust

#40

Post by itishappy »

galaxyLogic wrote: post #29
unoti wrote: post #17

Earlier quoted context omitted.

> Immutable data is not a “foundation of scalability and robustness”. It may not be the only way to get to scalability and robustness. But it certainly is the cornerstone of how Erlang gets there. 1. First, the way Erlang treats data ensures that every piece of data can be sent over the wire by default. This helps pave the way for another amazing characteristic of Erlang, and that is when you refer to and use an obje…

> you instantiate it you communicate with it by sending it messages Does this mean a web-endpoint has to be immutable? If you send it the same parameters multiple times, is it required to respond with the same response every time? If not, does that not mean it is in fact mutable? I read elsewhere that in Elixir programs, there is no difference in messaging a local "agent" or a remote one? The caller does not know whe…

> Does this mean a web-endpoint has to be immutable? If you send it the same parameters multiple times, is it required to respond with the same response every time? If not, does that not mean it is in fact mutable?

This is a concept called purity, and it's only loosely related to immutability. Immutability makes purity easier to implement and reason about, but does not guarantee it. Erlang/Elixer are not pure. For example, `DateTime.now("Etc/UTC")` will return different things at different times.

As a counter-example, Haskell functions are pure, so the `getCurrentTime` function cannot return a value directly as it would be different every time. Instead it uses the return type `IO UTCTime` which act like instructions on how to calculate the time, rather than the time itself.

https://en.wikipedia.org/wiki/Pure_function

Post reply on HN