Live data from Hacker News

Do messages get lost when Erlang modules are upgraded?

pankajmore.in

11–20 of 37 posts

Re: Do messages get lost when Erlang modules are upgraded?

#11
post #6

This is pretty far from what other frameworks and systems can even dream about. Not everyone needs this but when they do need it, I only know about Erlang that can handle it.

Well, I did Erlang-inspired runtime patching and even hot module updates (reloads) with Python and gevent.backdoor. However, it was a hack (loading new module and [non-atomically] replacing all references to old one), especially if compared to Erlang approach.

Re: Do messages get lost when Erlang modules are upgraded?

#12

Hot code reloading is really amazing in Erlang. We use it to patch a method without restarting on a regular basis. Getting reltool properly configured to deploy the new version of your application though is a MAJOR pain in the ass. for the non-erlangers: the 'simple' way (ie i have a tiny change i want to make, so i'll just ssh in and do it) is to just edit your code, then attach to an existing node, and do make:all(…

Do you have some sort of guidelines when and when not to do hot release upgrade?

I still have to find anybody shipping whole updatable releases with erlang - couchdb, rabbitmq, any big open source project, nobody is doing it. I think it's a bit overrated, because realistically nobody will want to do this in production, unless it is audited and tested in every possible way. If you have to update external resources with the code_change callback it can fail miserably. I have no problem using it in production for small changes that I fully understand, but big release - wouldn't do it. Too many things can go wrong at the same time to react. You have to audit and test the upgrade and the rollback, which means extra code in code_change to handle the rollback too. Quite heavy.

Re: Do messages get lost when Erlang modules are upgraded?

#13
In my very early years when discovering coding I remember fantasizing about a system where I could start my creation as very simple endless loop and add to it without having to stop it what ever would make up an application (... more a coding experiment in logo or basic it was at that time ;)

Fast forward almost 30 years: in Erlang light-weight processes are (most of the time) tail recursive functions handling messages... endless loops. Those light-weight processes are running those endless loops.

The described module upgrade functionality allows for uninterrupted system upgrades of servers/services in the back-end which btw. is often serious business in production and the cause of the reltool complexities.

BUT: the same hot code reloading system allows for very sleek development experience. You start your first version of the service and from there on update the source and the service changes its behavior most of the time with no restart, no lost state etc. (with the help of some monitoring tools source changes can be picked up automagically...)

This kind of dev-environment is simply flow-inducing.

Re: Do messages get lost when Erlang modules are upgraded?

#14
post #8

Earlier quoted context omitted.

How? You have a C++ or Java object instance running in a process how do you upgrade that code without restarting the process?

According to the wiki the OpenJDK supports Hotswap https://wiki.openjdk.java.net/display/mlvm/HotSwap . As for C or C++ it is platform specific, but shared objects and DLLs have been runtime loadable/unloadable for as long as I can remember. Building a hot swap feature on top of it would be challenging but doable.

The tricky part is transferring state between hot reloads, and that's part of what Erlang was designed around. State has to be passed in and owned by the callee. And if the format that state is stored in changes, you need to handle that translation in the new version.

It's definitely doable in C and C++. The question is whether it is easier to re-implement a bunch of what Erlang does for you, or to just use something that supports it directly.

Good ol' "build or buy?"

Re: Do messages get lost when Erlang modules are upgraded?

#15
post #6

This is pretty far from what other frameworks and systems can even dream about. Not everyone needs this but when they do need it, I only know about Erlang that can handle it.

Wouldn't a reliable message queue + a script execution engine be enough?

Re: Do messages get lost when Erlang modules are upgraded?

#16

Hot code reloading is really amazing in Erlang. We use it to patch a method without restarting on a regular basis. Getting reltool properly configured to deploy the new version of your application though is a MAJOR pain in the ass. for the non-erlangers: the 'simple' way (ie i have a tiny change i want to make, so i'll just ssh in and do it) is to just edit your code, then attach to an existing node, and do make:all(…

Could you talk a little bit about what happens to processes which are running different versions of the code? Do you make your code backward compatible so that old processes continue to work with the new process? Or do you just let the old processes crash?

Re: Do messages get lost when Erlang modules are upgraded?

#17

Hot code reloading is really amazing in Erlang. We use it to patch a method without restarting on a regular basis. Getting reltool properly configured to deploy the new version of your application though is a MAJOR pain in the ass. for the non-erlangers: the 'simple' way (ie i have a tiny change i want to make, so i'll just ssh in and do it) is to just edit your code, then attach to an existing node, and do make:all(…

Do you have some sort of guidelines when and when not to do hot release upgrade? I still have to find anybody shipping whole updatable releases with erlang - couchdb, rabbitmq, any big open source project, nobody is doing it. I think it's a bit overrated, because realistically nobody will want to do this in production, unless it is audited and tested in every possible way. If you have to update external resources wit…

Yes, I could not find any examples. Would be great to find out real world examples of people using erlang's hot code reloading in production.

Re: Do messages get lost when Erlang modules are upgraded?

#18
post #6

This is pretty far from what other frameworks and systems can even dream about. Not everyone needs this but when they do need it, I only know about Erlang that can handle it.

Wouldn't a reliable message queue + a script execution engine be enough?

How would a "reliable message queue + a script engine" be equivalent to a dynamically hot swappable code reloading? Could you elaborate?

Re: Do messages get lost when Erlang modules are upgraded?

#19

Hot code reloading is really amazing in Erlang. We use it to patch a method without restarting on a regular basis. Getting reltool properly configured to deploy the new version of your application though is a MAJOR pain in the ass. for the non-erlangers: the 'simple' way (ie i have a tiny change i want to make, so i'll just ssh in and do it) is to just edit your code, then attach to an existing node, and do make:all(…

Do you have some sort of guidelines when and when not to do hot release upgrade? I still have to find anybody shipping whole updatable releases with erlang - couchdb, rabbitmq, any big open source project, nobody is doing it. I think it's a bit overrated, because realistically nobody will want to do this in production, unless it is audited and tested in every possible way. If you have to update external resources wit…

Mochi Media uses hot code loading in production.

Re: Do messages get lost when Erlang modules are upgraded?

#20
I was a little confused by the title as it implied a relationship between a process's message box and some module which is not the case.

The article does at least demonstrate that processes can transition between two versions of the same code w/o resetting state, which is, at its core, the very thing that makes code upgrades remotely practical.

Other comments mention some more sophisticated machinery like release upgrades. Erlang also has many code upgrade options baked into the OTP as well. I make use of many of these features both during development but also in production with some careful review. I'm always disappointed when going back to a system that has to "reboot" itself after getting used to hot upgrades and distributed erlang (version discrepancies in a cluster can present a similar problem if you don't want to pause your system).

Post reply on HN