Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do messages get lost when Erlang modules are upgraded?

pankajmore.in

Post reply on HN
Print view View original

21–30 of 37 posts

Re: Do messages get lost when Erlang modules are upgraded?

#21

Post by strmpnk »

blankverse wrote: post #16
rubyrescue wrote: post #2

Hot code reloading is really amazing in Erlang. We use it to patch a method without restarting on a regular basis. Getting reltool properly configured to deploy the new version of your application though is a MAJOR pain in the ass. for the non-erlangers: the 'simple' way (ie i have a tiny change i want to make, so i'll just ssh in and do it) is to just edit your code, then attach to an existing node, and do make:all(…

Could you talk a little bit about what happens to processes which are running different versions of the code? Do you make your code backward compatible so that old processes continue to work with the new process? Or do you just let the old processes crash?

You can upgrade state in many cases if the changes are trivial, though depending on the ephemerality of a process you might let it finish or crash and restart instead of doing this.

Re: Do messages get lost when Erlang modules are upgraded?

#22

Post by rdtsc »

alanning wrote: post #9
rdtsc wrote: post #8

Earlier quoted context omitted.

How? You have a C++ or Java object instance running in a process how do you upgrade that code without restarting the process?

I imagine you would do what erlang does for you, just manually - isolate individual server instances, do the update, then make them public again. Say for example you are updating code in your service's web-tier. Have your LBs not send any more new traffic to the server instance, wait some reasonable length of time until existing connections are completed, deploy, restart, give LBs the A-OK to start traffic up again.…

Ok but what if that server you isolated was holding a long run process. Yes for a web service that serves quick http request and responses you could do that. Not everything is short lived request and response messages. Some sessions and processes are long lived. So you have a socket open and data streaming into it, it is not easy to isolate it. You could say send it a message that says -- start isolating.

How do you hotswap with data held in the stack or the heap? Even more if two part of your code need updated instance data, how does it ensure that update is synchronous or happens in the right order? In Java they have a $transformer() method. Ok how do you regulate the order in which $transformer() gets called. Otherwise a new version of one instance will call an old version of another.

I am not saying it is impossible to do it, there might be a way, but it is just usually working against the framework and against the default setup of the system.

Re: Do messages get lost when Erlang modules are upgraded?

#23

Post by blankverse »

strmpnk wrote: post #20

I was a little confused by the title as it implied a relationship between a process's message box and some module which is not the case. The article does at least demonstrate that processes can transition between two versions of the same code w/o resetting state, which is, at its core, the very thing that makes code upgrades remotely practical. Other comments mention some more sophisticated machinery like release upg…

The title is confusing! It should have been "Do messages get lost during code reloading in erlang?"

In a prod env, how do you make sure that different versions of your code coexist peacefully?

Re: Do messages get lost when Erlang modules are upgraded?

#24

Post by blankverse »

strmpnk wrote: post #21
blankverse wrote: post #16

Earlier quoted context omitted.

Could you talk a little bit about what happens to processes which are running different versions of the code? Do you make your code backward compatible so that old processes continue to work with the new process? Or do you just let the old processes crash?

You can upgrade state in many cases if the changes are trivial, though depending on the ephemerality of a process you might let it finish or crash and restart instead of doing this.

So, in the worst case, for achieving basic code reloading in other erlang like systems(ex.- cloud haskell), we could let every process in the cluster crash and restart with the new version?

Re: Do messages get lost when Erlang modules are upgraded?

#25

Post by saurik »

blankverse wrote: post #16
rubyrescue wrote: post #2

Hot code reloading is really amazing in Erlang. We use it to patch a method without restarting on a regular basis. Getting reltool properly configured to deploy the new version of your application though is a MAJOR pain in the ass. for the non-erlangers: the 'simple' way (ie i have a tiny change i want to make, so i'll just ssh in and do it) is to just edit your code, then attach to an existing node, and do make:all(…

Could you talk a little bit about what happens to processes which are running different versions of the code? Do you make your code backward compatible so that old processes continue to work with the new process? Or do you just let the old processes crash?

If you set things up right, the old process is upgraded to use the new code upon processing the next message (and has an opportunity to upgrade its state from the state the old code was using to the state the new code was using, if required); like, the idea is that you are trying to upgrade the running code. (In fact, leaving old processes lying around is actually problematic as you can only have two versions of any module loaded, the obsolete one and the current one; if you upgrade again all processes running the old version crash.)

Re: Do messages get lost when Erlang modules are upgraded?

#26

Post by cdcarter »

blankverse wrote: post #18
_random_ wrote: post #15

Earlier quoted context omitted.

Wouldn't a reliable message queue + a script execution engine be enough?

How would a "reliable message queue + a script engine" be equivalent to a dynamically hot swappable code reloading? Could you elaborate?

The message queue delivers messages to the script engine, running the script new every time. No code is running hot to be swapped into. At the next message, the new code is loaded. Similar to what Erlang does, but much more brittle, less tested, and more fail-prone.

Re: Do messages get lost when Erlang modules are upgraded?

#27

Post by strmpnk »

blankverse wrote: post #24
strmpnk wrote: post #21

Earlier quoted context omitted.

You can upgrade state in many cases if the changes are trivial, though depending on the ephemerality of a process you might let it finish or crash and restart instead of doing this.

So, in the worst case, for achieving basic code reloading in other erlang like systems(ex.- cloud haskell), we could let every process in the cluster crash and restart with the new version?

It's rarely "only crash" as that eventually bubbles up to being equivalent to a reboot. It's simply that you can have subsystems that do that in effective isolation if the effects of a reboot are minimal.

On the code state upgrade side there are usually a few different issues that can arise and I'd be very curious to hear if there are ways Haskell would handle some of these.

The first one is type changes. I might have a record that has a new field added. Now it's not necessarily pretty to upgrade on call with a pattern match or using a code upgrade protocol but it's easily expressed dynamically.

Another is in the interface, like adding new arguments or changing from a synchronous call to an asynchronous one. These are a bit easier to handle via indirection though they show that you'll need to plan your entry/exit points for upgrades carefully (again, OTP has things like gen_server which make this much easier).

If Haskell can manage to get past they type boundary issue then it's really a matter of supporting at least 2 simultaneous versions of code so each process can be scheduled and upgraded in natural course. Handling more than 2 could be of use depending on how aggressively you want to purge, for example, a local rather than fully qualified call can be caught in a closure and passed around as in some value to be called later. These long lived references will need to be handled carefully or you might get some delayed surprises.

Re: Do messages get lost when Erlang modules are upgraded?

#28

Post by strmpnk »

blankverse wrote: post #23
strmpnk wrote: post #20

I was a little confused by the title as it implied a relationship between a process's message box and some module which is not the case. The article does at least demonstrate that processes can transition between two versions of the same code w/o resetting state, which is, at its core, the very thing that makes code upgrades remotely practical. Other comments mention some more sophisticated machinery like release upg…

The title is confusing! It should have been "Do messages get lost during code reloading in erlang?" In a prod env, how do you make sure that different versions of your code coexist peacefully?

Related to other comments, there are some mechanisms that ensure that two versions of all code can remain active (if you chose) in Erlang. No extra effort is required.

In terms of correctness, I'm not aware of any work that does type checking or serious analysis across version upgrades. Even Erlang's dialyzer will only check each version in isolation. So in practice this means you either test the upgrades with things like continuous integration or even manually, OR do so in very careful and small steps that are easy to reason about locally (very useful for critical applications that can't spare downtime).

Re: Do messages get lost when Erlang modules are upgraded?

#29

Post by banachtarski »

blankverse wrote: post #23
strmpnk wrote: post #20

I was a little confused by the title as it implied a relationship between a process's message box and some module which is not the case. The article does at least demonstrate that processes can transition between two versions of the same code w/o resetting state, which is, at its core, the very thing that makes code upgrades remotely practical. Other comments mention some more sophisticated machinery like release upg…

The title is confusing! It should have been "Do messages get lost during code reloading in erlang?" In a prod env, how do you make sure that different versions of your code coexist peacefully?

You can have multiple sets of instructions for the same function but indicate that only one should be active. This makes it easy to rollback a deploy for example. Also, when you do a live upgrade, there must be at least two versions exposed to the VM at one point before the switch occurs.

Easy hot code reloading is one of the great benefits from CSP.

Re: Do messages get lost when Erlang modules are upgraded?

#30

Post by yuubi »

blankverse wrote: post #16
rubyrescue wrote: post #2

Hot code reloading is really amazing in Erlang. We use it to patch a method without restarting on a regular basis. Getting reltool properly configured to deploy the new version of your application though is a MAJOR pain in the ass. for the non-erlangers: the 'simple' way (ie i have a tiny change i want to make, so i'll just ssh in and do it) is to just edit your code, then attach to an existing node, and do make:all(…

Could you talk a little bit about what happens to processes which are running different versions of the code? Do you make your code backward compatible so that old processes continue to work with the new process? Or do you just let the old processes crash?

If you use OTP (gen_server, etc), your code is a set of callbacks that return promptly, so it never stays "old" until the next upgrade. Your callbacks produce and use a "state" object (opaque to OTP) that OTP passes around[1], and OTP calls a code_change method whenever your module changes, so you can change the format of this object between versions.

The gen_server itself passes its module name to sys:handle_system_msg[2] when it receives a system message like a code-change notification, which ends up being the Module:Function type of call that always refers to the new version of the module, so it always ends up running the latest version when you upgrade gen_server.

[1] http://www.erlang.org/doc/man/gen_server.html#Module:init-1

[2]file:///C:/Program%20Files/erl5.9.1/lib/stdlib-1.18.1/doc/html/sys.html#handle_system_msg-6

Post reply on HN