Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hot code reloading with Erlang

medium.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Hot code reloading with Erlang

#21

Post by simoncion »

jgalt212 wrote: post #17
simoncion wrote: post #11

Earlier quoted context omitted.

What -exactly- do you want to do when you say you want to restart the Erlang VM? I'm asking because I don't have enough context to know why you want to do what you're asking to do.

I would assume the longer any VM is running the higher the chances of a service degrading. I guess this is mostly due to memory leaks or bit rot. I have no Erlang VM experience, so my comment was geared towards VMs in general.

> I guess this is mostly due to memory leaks or bit rot.

"Bit rot"? The only defense for the bit rot I'm aware of is ECC RAM.

Anyway. AFAIK (and I'm no Erlang expert, so there's probably something pertinent that I don't know) unless there's a resource leak in core Erlang code, resource leaks can be fixed by restarting the leaking application, or killing the leaking process. [0]

[0] Erlang software is often broken up into Applications. [1] An application is a collection of code with a well-known entry point that (ideally) does a particular thing. An application can depend on other applications and the services that they provide. Applications can be started and stopped independently of all others in the system, but -in order to keep running- dependant applications need to be designed to handle the temporary absence of an application that they depend on.

[1] http://learnyousomeerlang.com/building-applications-with-otp

Re: Hot code reloading with Erlang

#22

Post by pmontra »

pmontra wrote: post #16
stevegh wrote: post #8

Interesting. The hot code reloading functionality in Erlang led me to investigate ruby (my preferred Dev language) a bit more. You can do a hot code load in Ruby using the Kernel#load() call. It won't alter functionality currently on the call stack, but it will change the functionality of everything not on the call stack. With some sympathetic design, you can achieve hot code loading fo high availability in ruby.

You can use that to replace code by monkey patching $ cat hi.rb def method puts "hi" end method load("hello.rb") method $ cat hello.rb def method puts "hello" end $ ruby hi.rb hi hello You must engineer your application to execute the load method and that's it. However I wonder if this is really equivalent to what Erlang does. I remember http://rvirding.blogspot.it/2008/01/virdings-first-rule-of-p...

Interesting post at http://blog.rkh.im/code-reloading

Re: Hot code reloading with Erlang

#23

Post by toast0 »

simoncion wrote: post #11
jgalt212 wrote: post #4

You know what would be really amazing is if you could restart the Erlgang VM, or load new a VM without interrupting any of the running code modules.

What -exactly- do you want to do when you say you want to restart the Erlang VM? I'm asking because I don't have enough context to know why you want to do what you're asking to do.

Not the original asker, but it would be nice to be able to upgrade between OTP releases, without having to restart my application (but I have no expectation of that being possible ever when there are VM changes, and unlikely for the effort to be spent to do it for the user-space erlang bits either). I have to use DNS for load balancing [1] and big mnesia tables, so I have to wait a long time for traffic to drain, and then another long time for the application to start back up.

Working for 4 years in an Erlang environment where hotloading is the norm, makes me wish for it everywhere! Why do I have to reboot to fix kernel bugs in tcp? :(

[1] the load balancers I have access to where we host had more downtime than our hosts, so not actually helpful

Re: Hot code reloading with Erlang

#24

Post by toast0 »

simoncion wrote: post #20
yetihehe wrote: post #18

Earlier quoted context omitted.

When your server has several gigs of state. It's VERY useful on a dev server. Instead of waiting several minutes for reload, I just load in new code manually (typically I change only 1-2 files per reload). If something breaks - hey, it's only dev server. Erlangs other feature - almost everything works alone - helps with this. If something breaks, it breaks only in one place, so most of the time I only need to make sm…

> Instead of waiting several minutes for reload, I just load in new code manually (typically I change only 1-2 files per reload). If something breaks - hey, it's only dev server. Someone wrote a module for elixir that uses inotify (and similar) to -I think- watch .beam files for modification and perform the required hot-reloads automatically. I would be reluctant to run this in production, and I can see situations (e…

This would be terrible in production -- often there's an order you need to load the beam files, and I wouldn't want to add that to the compile step, it's very simple to load in the proper order. You could pretty easily use code:soft_purge/1 prior to loading to avoid killing lingering processes though, and then it would probably be reasonable for development.

Re: Hot code reloading with Erlang

#25

Post by simoncion »

toast0 wrote: post #24
simoncion wrote: post #20

Earlier quoted context omitted.

> Instead of waiting several minutes for reload, I just load in new code manually (typically I change only 1-2 files per reload). If something breaks - hey, it's only dev server. Someone wrote a module for elixir that uses inotify (and similar) to -I think- watch .beam files for modification and perform the required hot-reloads automatically. I would be reluctant to run this in production, and I can see situations (e…

This would be terrible in production -- often there's an order you need to load the beam files, and I wouldn't want to add that to the compile step, it's very simple to load in the proper order. You could pretty easily use code:soft_purge/1 prior to loading to avoid killing lingering processes though, and then it would probably be reasonable for development.

> This would be terrible in production...

Yeah it could be. Frankly, I'd likely reach for Erlang Releases before I reached for this when updating software in production.

However, for a large variety of dev work, this automatic module reloading thingie works pretty well. :)

> You could pretty easily use code:soft_purge/1 prior to loading to avoid killing lingering processes though...

Mmm. Okay. So, I'm not 100% on how this works, so please bear with me and my inaccurate terminology. :(

In any given Erlang system, there can be two versions of a module running, the "current" one, and the "old" one, right?

So, if you call code:soft_purge/1 when there is no "old" code loaded, it should return true, yes? (In addition to returning true when there's no process running the "old" code.) [0]

So, would this be a way to write an auto-loader that doesn't purge in-use code?

* code:soft_purge(?MODULE)

* if false, wait a while then retry

* if true, code:load_file(?MODULE)

I guess maybe you'd want to build up a list of all the modules that have been modified, and wait until code:soft_purge/1 returns true for all of them before loading the modules. (maybe.)

You also -obviously- want an override that allows for the purging of in-use code.

[0] Testing indicates that it does, but it's often good to double-check. :)

Re: Hot code reloading with Erlang

#26

Post by simoncion »

toast0 wrote: post #23
simoncion wrote: post #11

Earlier quoted context omitted.

What -exactly- do you want to do when you say you want to restart the Erlang VM? I'm asking because I don't have enough context to know why you want to do what you're asking to do.

Not the original asker, but it would be nice to be able to upgrade between OTP releases, without having to restart my application (but I have no expectation of that being possible ever when there are VM changes, and unlikely for the effort to be spent to do it for the user-space erlang bits either). I have to use DNS for load balancing [1] and big mnesia tables, so I have to wait a long time for traffic to drain, and…

ISTR that, you can reload core Erlang modules, but that there's some sticky_directory stuff that prevents it from happening by default.

I'm pretty sure that I can reload the inet module: [0]

    Eshell V7.0  (abort with ^G)
    1>  l(inet).
    {error,sticky_directory}
    
    =ERROR REPORT==== 6-Dec-2015::03:37:00 ===
    Can't load module 'inet' that resides in sticky dir
    2> code:which(inet).
    "/usr/lib/erlang/lib/kernel-4.0/ebin/inet.beam"
    3> code:unstick_dir("/usr/lib/erlang/lib/kernel-4.0/ebin/"). 
    ok
    4> l(inet).
    {module,inet}
    5>
Not that this is a good idea, mind, but I'm fairly certain that it's doable. :)

(Also note that you can reload the mnesia module without hassle. Its ebin directory is not marked as sticky. :) )

> I have to use DNS for load balancing [because the load balancers fall over often].

Oh lord. That's a terrible situation to be in.

[0] Which is part of the kernel application, which is one of the applications that hot upgrades require that you restart the emulator to upgrade. [1]

[1] http://www.erlang.org/doc/system_principles/upgrade.html

Re: Hot code reloading with Erlang

#27

Post by toast0 »

simoncion wrote: post #25
toast0 wrote: post #24

Earlier quoted context omitted.

This would be terrible in production -- often there's an order you need to load the beam files, and I wouldn't want to add that to the compile step, it's very simple to load in the proper order. You could pretty easily use code:soft_purge/1 prior to loading to avoid killing lingering processes though, and then it would probably be reasonable for development.

> This would be terrible in production... Yeah it could be. Frankly, I'd likely reach for Erlang Releases before I reached for this when updating software in production. However, for a large variety of dev work, this automatic module reloading thingie works pretty well. :) > You could pretty easily use code:soft_purge/1 prior to loading to avoid killing lingering processes though... Mmm. Okay. So, I'm not 100% on how…

Yes, you've got the concepts and implementation correct.

The exact strategy for reloading (wait for all at once, load whatever is ready, how long to wait, etc), left as an exercise for the reader. For dev, I use a function in the shell that loads everything that changed (no soft purge), for prod, i have a function that goes in order and checks soft purge, then loads (if the 2nd module doesn't soft purge, it will have already loaded the first module, but it will stop before trying the 3rd).

With most things in gen_server's, there's not a lot of opportunity for lingering code, but sometimes it happens.

Re: Hot code reloading with Erlang

#28

Post by toast0 »

simoncion wrote: post #26
toast0 wrote: post #23

Earlier quoted context omitted.

Not the original asker, but it would be nice to be able to upgrade between OTP releases, without having to restart my application (but I have no expectation of that being possible ever when there are VM changes, and unlikely for the effort to be spent to do it for the user-space erlang bits either). I have to use DNS for load balancing [1] and big mnesia tables, so I have to wait a long time for traffic to drain, and…

ISTR that, you can reload core Erlang modules, but that there's some sticky_directory stuff that prevents it from happening by default. I'm pretty sure that I can reload the inet module: [0] Eshell V7.0 (abort with ^G) 1> l(inet). {error,sticky_directory} =ERROR REPORT==== 6-Dec-2015::03:37:00 === Can't load module 'inet' that resides in sticky dir 2> code:which(inet). "/usr/lib/erlang/lib/kernel-4.0/ebin/inet.beam"…

Yes, you can reload all the beam files (even if you have to unsticky them -- you can do that), but that doesn't mean r16 -> r17 didn't make changes you can't safely load. Some parts of OTP have state upgraders, and some don't; some may depend on bif/nif changes as well.

>> I have to use DNS for load balancing [because the load balancers fall over often].

>Oh lord. That's a terrible situation to be in.

It's not terrible, it's just not great. Our hosting environment is generally very reliable, so if I don't screw things up, my systems won't fall over. It's just their loadbalancers are crap, the suggested upgrade path was run a load balancer in a VM appliance; which seems like maybe I should just run CARP myself on the hosts (or something), instead, and skip a layer, but I'll probably never get around to that, because it doesn't come up that often :)

Re: Hot code reloading with Erlang

#29

Post by simoncion »

toast0 wrote: post #28
simoncion wrote: post #26

Earlier quoted context omitted.

ISTR that, you can reload core Erlang modules, but that there's some sticky_directory stuff that prevents it from happening by default. I'm pretty sure that I can reload the inet module: [0] Eshell V7.0 (abort with ^G) 1> l(inet). {error,sticky_directory} =ERROR REPORT==== 6-Dec-2015::03:37:00 === Can't load module 'inet' that resides in sticky dir 2> code:which(inet). "/usr/lib/erlang/lib/kernel-4.0/ebin/inet.beam"…

Yes, you can reload all the beam files (even if you have to unsticky them -- you can do that), but that doesn't mean r16 -> r17 didn't make changes you can't safely load. Some parts of OTP have state upgraders, and some don't; some may depend on bif/nif changes as well. >> I have to use DNS for load balancing [because the load balancers fall over often]. >Oh lord. That's a terrible situation to be in. It's not terrib…

> ...but that doesn't mean r16 -> r17 didn't make changes you can't safely load.

Oh yeah. I know. I would expect for some-to-many minor things (like the TCP bug you mentioned), this wouldn't be an insane way to do a hot upgrade. Guess one would need to make a close study of the diffs.

Regardless, I mentioned the fact that this isn't a good idea for any novice who might stumble across this comment months or years hence and get the wrong idea. :)

> ...maybe I should just run CARP myself on the hosts...

Oh, CARP is so cool. It's a bit of a pity that ucarp doesn't support IPv6. But, how does CARP replace a load balancer? Isn't it used for single host availability, or is my understanding too narrow?

Re: Hot code reloading with Erlang

#30

Post by toast0 »

simoncion wrote: post #29
toast0 wrote: post #28

Earlier quoted context omitted.

Yes, you can reload all the beam files (even if you have to unsticky them -- you can do that), but that doesn't mean r16 -> r17 didn't make changes you can't safely load. Some parts of OTP have state upgraders, and some don't; some may depend on bif/nif changes as well. >> I have to use DNS for load balancing [because the load balancers fall over often]. >Oh lord. That's a terrible situation to be in. It's not terrib…

> ...but that doesn't mean r16 -> r17 didn't make changes you can't safely load. Oh yeah. I know. I would expect for some-to-many minor things (like the TCP bug you mentioned), this wouldn't be an insane way to do a hot upgrade. Guess one would need to make a close study of the diffs. Regardless, I mentioned the fact that this isn't a good idea for any novice who might stumble across this comment months or years henc…

Oh, I meant a os kernel bug; no way to hotload those.

I have two servers, if I get two carpable ips, and may each server primary on one, and put them both in DNS, I have load balancing and failover. If I just use one IP, at least I have failover, anyway I need each server to be able to handle full load, so I could have hot-warm instead of hot-hot.

(with more than two servers, need to figure something else out, probably two boxes running a simple load balancing in front of the rest of the cluster.

Post reply on HN