Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elixir/Erlang Hot Swapping Code (2016)

kennyballou.com

Post reply on HN
Print view View original

91–100 of 119 posts

Re: Elixir/Erlang Hot Swapping Code (2016)

#91

Post by knome »

Muromec wrote: post #17
anonymousDan wrote: post #8

I'm a distributed setup I imagine there could be cases where you want to atomically hot upgrade multiple VMs at the same time. Is this common in practice and if so are there recommended patterns/techniques for doing it?

There can't be anything atomic in a distributed system. You can't even atomically hot upgrade it on a single VM anyway -- you instead load the new version of the module and let dispatcher know to route new calls into it, the same as you would do with a load balancer and a bunch of load bearing docker hosts, just inside your app.

erlang has a code_change function in the otp that allows the gen_server to update its current state and start using new code. No connections need be broken with clients, no long running processes need be stopped. Just updated in place.

It's not just a routing change.

https://www.erlang.org/docs/24/man/gen_server

Re: Elixir/Erlang Hot Swapping Code (2016)

#92

Post by cess11 »

cess11 wrote: post #78
modernerd wrote: post #75

Live updating a drone running Erlang in 10ms while it was flying with no application restart and no loss of state impressed me when I saw it in 2021: https://www.youtube.com/watch?v=XQS9SECCp1I But I almost never hear Erlang/Elixir/Gleam folks talk about this benefit of the Erlang VM now, even though it seems fairly unique and interesting. Has the community moved away from it? Is it just not that useful?

A lot of the GenServer-information floating around explains code_change/3, no? That's commonly what you want, a way to handle state propagation when process code is updating in a running system. Most people are probably running some web services or something and might as well shift machines in and out of a cluster or can wait for old processes to disband on their own, because the new code is backwards compatible with…

Someone put a reply and then deleted it while I wrote a response, and it irks me that it might have been a waste so here's the gist of it:

"Is it just that people are more comfortable with blue-green deploys, or are blue-green deploys actually better?"

It depends. If you can do a blue-green shift where you gradually add 'fresh' servers/VM:s/processes and drain the old, that's likely to be most convenient and robust in many organisations. On the other hand, if you rely on long running processes in a way where changing their PID:s break the system, then you pretty much need to update them with this kind of hot patching.

"Does Erlang offer any features to minimize damage here?"

The BEAM allows a lot of things in this area, on pretty much every level of abstraction. If you know what you're doing and you've designed your system to fit well into the provided mechanisms the platform provides a lot of support for hot patching without sacrificing robustness and uptime. But it's like another layer of possible bugs and risks, it's not just your usual network and application logic that might cause a failure, your handling of updates might itself be a source of catastrophe.

In practice you need to think long and hard about how to deploy, and test thoroughly under very production like conditions. It helps that you can know for sure what production looks like at any given time, the BEAM VM can tell you exactly what processes it runs, what the application and supervisor trees look like, hardware resource consumption and so on. You can use this information to stage fairly realistic tests with regards to load and whatnot, so if your update for example has an effect on performance and unexpected bottlenecks show up you might catch it before it reaches your users.

And as anyone can tell you who has updated a profitable, non-trivial production system directly, like a lot of PHP devs of ye olden times, it takes a rather strong stomach even when it works out fine. When it doesn't, you get scars that might never fade.

Re: Elixir/Erlang Hot Swapping Code (2016)

#94

Post by chamomeal »

behnamoh wrote: post #13

Lisp has had this features since day 1. But Lisp-like langs like Clojure, Racket, etc. don't have it. This is one of the fundamental features of Common Lisp and I don't know why most other Lisp-wanna-be's don't implement it.

Can’t you do something like this with clojure?

I don’t understand the particulars, but one selling point of biff is it’s got built-in support for updating things directly in prod via the REPL.

There’s a fun interview with the biff guy on the podcast “the REPL”. He talks about how much fun it is to develop directly on the prod server, and how horrified people are by it lol.

https://biffweb.com/

https://www.therepl.net/episodes/48/

Re: Elixir/Erlang Hot Swapping Code (2016)

#95

Post by Muromec »

cess11 wrote: post #78
modernerd wrote: post #75

Live updating a drone running Erlang in 10ms while it was flying with no application restart and no loss of state impressed me when I saw it in 2021: https://www.youtube.com/watch?v=XQS9SECCp1I But I almost never hear Erlang/Elixir/Gleam folks talk about this benefit of the Erlang VM now, even though it seems fairly unique and interesting. Has the community moved away from it? Is it just not that useful?

A lot of the GenServer-information floating around explains code_change/3, no? That's commonly what you want, a way to handle state propagation when process code is updating in a running system. Most people are probably running some web services or something and might as well shift machines in and out of a cluster or can wait for old processes to disband on their own, because the new code is backwards compatible with…

This is also a reply to that deleted comment, because I had to type it all and also got to go outside and have my European 2 hour long lunch break while doing it.

If you have any kind of state in gen_server and the state or assumptions of it have changed, you need to write that code_change thingy that migrates the state both ways between two specific versions. If by some chance this function is bugged, then the process is killed (which is okay), so you need to nail down the supervision tree to make things restartable and also not get into restart loops. Remember writing database migrations for django or whatever ORM of the day? Now do that, but for memory structures you have.

Now, while the function is running it can't be updated of course, so you need gen_server to call you back from the outside of the module. If you like to save function references instead of saving process references in your state, you need to figure out which version you will be actually calling.

If you change the arity of your record, then the old record no longer matches your patterns.

Since updates are not atomic, you will have two versions of the code running at the same time, potentially sending messages that old/new stuff does not expect, and both old and new code should not bug out. And if they do bug out, you have been smart enough to figure out how to recover and actually test that.

Than there is this thing, if somehow something from the version V-2 still running after update to V-1 and you start updating to the latest V, then things happen.

You can deal with all that of course and erlang gives you tools and recipies to make it work. Sometimes you have to make it work, because restarting and losing state is not an option. Also it's probably fun to deal with complex things.

Or you could just do do the stupid thing that is good enough and let it crash and restart instead of figuring out ten different things that could go wrong. Or take a 15 minutes maintenance window while your users are all sleeping (yes, not everybody is doing critical infra that runs 24/7 like discord group with game memes). Or just do blue-green and sidestep it all completely.

Re: Elixir/Erlang Hot Swapping Code (2016)

#96

Post by osmano807 »

lamuswawir wrote: post #34
behnamoh wrote: post #13

Lisp has had this features since day 1. But Lisp-like langs like Clojure, Racket, etc. don't have it. This is one of the fundamental features of Common Lisp and I don't know why most other Lisp-wanna-be's don't implement it.

Came here to say this. In Lisp, you can just compile a function, or load a file and it just works. It's not even sold as a hot feature, not the way Erlang sells it. It's just a feature. I manage a few websites written in Lisp, and updating them is as simple as push code, recompile and it works.

Those sites have something like Phoenix LiveView or it's something ad hoc like a simple SSR template engine? Would be nice to have something to handle migrations in the client side code to match the server side API.

Re: Elixir/Erlang Hot Swapping Code (2016)

#97

Post by Muromec »

knome wrote: post #91
Muromec wrote: post #17

Earlier quoted context omitted.

There can't be anything atomic in a distributed system. You can't even atomically hot upgrade it on a single VM anyway -- you instead load the new version of the module and let dispatcher know to route new calls into it, the same as you would do with a load balancer and a bunch of load bearing docker hosts, just inside your app.

erlang has a code_change function in the otp that allows the gen_server to update its current state and start using new code. No connections need be broken with clients, no long running processes need be stopped. Just updated in place. It's not just a routing change. https://www.erlang.org/docs/24/man/gen_server

It's a routing change in a sense that gen_server is routing function calls to the new module definition. I know about gen_server and code_change, the point was that conceptually the same mechanism, just on a different level of abstraction.

Re: Elixir/Erlang Hot Swapping Code (2016)

#98

Post by Muromec »

yetihehe wrote: post #89
Muromec wrote: post #87

Earlier quoted context omitted.

Do you do 100% of deployments using hot reload? If yes, maybe you should share the recipy with everybody else, since consensus seems to recommend the opposite. At the very least you will have a different procedure to upgrade the erlang itself, right? >If you have something simple, you don't need to use erlang I think on a spectrum of difficult things there is an area between hosting static file on rpi at home and run…

> Do you do 100% of deployments using hot reload? About 99%. We need to restart servers maybe once a year. Maybe next year we will finally migrate from erlang 21 to latest. Most "stopping everything" deployments take max 1s of downtime, like this month when we needed to upgrade postgresql database by switching it over to new machine, having zero downtime here would take a little longer to get the consistency, but we…

Sounds like you are doing cool stuff the cool way, which can't be said about everyone.

>Is PHP good for both? I think PHP is NOT good for long running stateful processes, but I didn't use it in 10years. And it probably is not needed for static files.

No, of course it isn't. I didn't touch it with a long pole for even more years and don't even want to. And would not argue to do what you are doing in anything that isn't running on BEAM/OTP.

The point I'm trying to make is -- for most of the web stuff, making transactional response from a short-lived http handler is good enough and you can do it even in PHP (which is not a praise for PHP as a great tool I enjoy using, but the opposite). It would not the most optimal solution by any metric, not the most elegant or sophisticated either, but it's survivable, it's the lowest bidder.

Re: Elixir/Erlang Hot Swapping Code (2016)

#99

Post by Muromec »

thanksgiving wrote: post #73
Muromec wrote: post #32

Earlier quoted context omitted.

Sounds like what happened with hot upgrade privileges in some erlang shops too.

I've worked at a smaller project where it worked just fine. The key I think is to have the project be small enough to be able to fit in my head. That and have an identical test server. I used to make changes locally, test it locally, then make the same change on test, have someone else look at it, get a lgtm, and do the same thing on the production machine. It sounds like a lot of steps but it is pretty straightforwa…

A lot of things work on a small project where everybody knows what they are doing and can keep all the dependencies and data structure in their heads. Logging to the production server and editing PHP file right where apache is looking at it and fixing stuff with sql commands in the production database to address customer complaints.

The big question is what everyone else should be doing that survives the touch with unevenly distributed amounts of technical expertise and amount fucks given about result.

Re: Elixir/Erlang Hot Swapping Code (2016)

#100

Post by epiccoleman »

I wonder if this kind of thing could be used to make the Elixir REPL a bit more LISPy. I like iex a good deal, but I often find myself wishing I could just easily eval some code or expression in the editor and have it make its way into the REPL context. (yes, I know you can `r` on a module, but that's pretty clunky compared to something like CIDER).
Post reply on HN