Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elixir/Erlang Hot Swapping Code (2016)

kennyballou.com

Post reply on HN
Print view View original

21–30 of 119 posts

Re: Elixir/Erlang Hot Swapping Code (2016)

#21

Post by AlphaWeaver »

anonymousDan wrote: post #8

I'm a distributed setup I imagine there could be cases where you want to atomically hot upgrade multiple VMs at the same time. Is this common in practice and if so are there recommended patterns/techniques for doing it?

Erlang does have a mechanism that allows a module to control when it moves from the "old version" to the "new version" of its own code. Calls to the module with the fully qualified name (e.g. `module:function()`) will invoke the "new code" once it's loaded, but calls within that module using only function names (just `function()`) will continue to invoke the "old code".

If the portion of the app you were hot upgrading was an OTP process like a GenServer, you could theoretically wait for some sort of atomic coordination mechanism to make that fully qualified function call after the new code has loaded, at least in theory.

We use hot code reloading at my work, but haven't had a reason to atomically sync the reload. Most of the time it's a tmux session with `synchronize-panes` and that suffices. If your application can handle upgrades within a module smoothly, it's rare to have a need for some sort of cluster-level coordination of a code change, at least one that's atomic.

Re: Elixir/Erlang Hot Swapping Code (2016)

#22

Post by zanderwohl »

saurik wrote: post #4
omertoast wrote: post #2

i'm so sick of this DevOps bullshit i wonder if there is an alternative language that you can hot swap code and do all the black magic stuff while keeping the reliability and performance like Rust.

I am shocked at the idea of anyone implying that Erlang is "unreliable"... it's entire reason for existence was to step up the game on reliability.

I agree. I don't know much about Erlang but what I've heard seems to indicate it's used for high-uptime systems that handle errors well.

Re: Elixir/Erlang Hot Swapping Code (2016)

#23

Post by AnotherGoodName »

diath wrote: post #16
slt2021 wrote: post #5

hot reload of code is nothing new nowadays, but people use it only locally during development for REPL like development style. in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container

> in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container How do you think video games like World of Warcraft or Path of Exile deploy restartless hotfixes to millions of concurrent players without killing instances? I don't think it's a matter of "prefer to", it's a matter of "can we completely disrupt the service for users and pote…

That’s a very big assumption that they do code hotpatching.

It would seem far more likely they seperate the stateful (database) and stateless layers (game logic) and they just spin up a new instance of the stateless server layer behind a reverse proxy and spin down the old instance. It’s basically how all websites update without down time.

Re: Elixir/Erlang Hot Swapping Code (2016)

#24

Post by AlphaWeaver »

slt2021 wrote: post #5

hot reload of code is nothing new nowadays, but people use it only locally during development for REPL like development style. in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container

People may "prefer" simply replacing containers, but as some siblings mention, some applications might require more reliability guarantees.

Erlang was originally designed for implementing telephony protocols, where interrupted phone calls were not an acceptable side effect of application updates.

Re: Elixir/Erlang Hot Swapping Code (2016)

#25

Post by diath »

qudat wrote: post #20
diath wrote: post #16

Earlier quoted context omitted.

> in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container How do you think video games like World of Warcraft or Path of Exile deploy restartless hotfixes to millions of concurrent players without killing instances? I don't think it's a matter of "prefer to", it's a matter of "can we completely disrupt the service for users and pote…

WoW restarts every week. Not sure that’s better than zero downtime deployments

That's just how it works when your backend is a hybrid software that utilizes a low-level compiled programming language and a high-level language that runs in its own VM. You can use the latter for gameplay features, and can hotfix on the go, and then for core changes you have to restart, which is also why WoW will hotfix the latter on the go, usually every day on an expansion launch, whereas they defer the bulk of backend changes for the next weekly restart without continuously disrupting the game for players.

Re: Elixir/Erlang Hot Swapping Code (2016)

#26

Post by AnotherGoodName »

AlphaWeaver wrote: post #24
slt2021 wrote: post #5

hot reload of code is nothing new nowadays, but people use it only locally during development for REPL like development style. in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container

People may "prefer" simply replacing containers, but as some siblings mention, some applications might require more reliability guarantees. Erlang was originally designed for implementing telephony protocols, where interrupted phone calls were not an acceptable side effect of application updates.

FWIW as soon as you start using containers you should be able to handle those containers spinning up/down. Pretty much the whole point of containers. At which point you don’t need to bother with code hot swapping since you already have a mechanism for newer containers to spin up while older ones spin down.

The sibling post “that’s how they update without downtime” is super naive. It is absolutely not how they do it.

Re: Elixir/Erlang Hot Swapping Code (2016)

#27

Post by diath »

AnotherGoodName wrote: post #23
diath wrote: post #16

Earlier quoted context omitted.

> in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container How do you think video games like World of Warcraft or Path of Exile deploy restartless hotfixes to millions of concurrent players without killing instances? I don't think it's a matter of "prefer to", it's a matter of "can we completely disrupt the service for users and pote…

That’s a very big assumption that they do code hotpatching. It would seem far more likely they seperate the stateful (database) and stateless layers (game logic) and they just spin up a new instance of the stateless server layer behind a reverse proxy and spin down the old instance. It’s basically how all websites update without down time.

A website that just proxies to another server does not need to do much to restore the previous state to make it look seamless to a user, the client will just perform another GET request that triggers a few SELECT queries, it's far more complex in the context of a video game.

Re: Elixir/Erlang Hot Swapping Code (2016)

#28

Post by Thaxll »

diath wrote: post #16
slt2021 wrote: post #5

hot reload of code is nothing new nowadays, but people use it only locally during development for REPL like development style. in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container

> in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container How do you think video games like World of Warcraft or Path of Exile deploy restartless hotfixes to millions of concurrent players without killing instances? I don't think it's a matter of "prefer to", it's a matter of "can we completely disrupt the service for users and pote…

Most of those hot fixes are data driven as in database updates. Gameserver just reload the data, the binary itself is not touch.

I've never seen a game where they hot reload code inside the gameserver itself, it's usually a downtime or rolling updates.

Re: Elixir/Erlang Hot Swapping Code (2016)

#29

Post by diath »

Thaxll wrote: post #28
diath wrote: post #16

Earlier quoted context omitted.

> in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container How do you think video games like World of Warcraft or Path of Exile deploy restartless hotfixes to millions of concurrent players without killing instances? I don't think it's a matter of "prefer to", it's a matter of "can we completely disrupt the service for users and pote…

Most of those hot fixes are data driven as in database updates. Gameserver just reload the data, the binary itself is not touch. I've never seen a game where they hot reload code inside the gameserver itself, it's usually a downtime or rolling updates.

> Most of those hot fixes are data driven as in database updates. Gameserver just reload the data, the binary itself is not touch.

And since the data from the disk/database (whether it's a Lua table, XML structure, JSON object, or a query) is then representend as a low-level data structure, that's essentially what hot reloading is - you deserialize the new data and hot-swap the pointers in the simplest terms.

>I've never seen a game where they hot reload code inside the gameserver itself, it's usually a downtime or rolling updates.

In World of Warcraft, you will literally have bosses despawn mid-fight and spawn again with new stats or you will see their health values update mid-fight, all without the players getting interrupted, their spell state getting desynced, or spawned items in the instance disappearing. This can be observed with the release of every single new raid on live streams as Blizzard employees are watching the world first attempts and tweaking/tuning the fights as they happen.

EDIT: Here's such an example, for the majority of the fight the extra tank could keep a spawned monster away from the boss, then mid-fight, the monster suddenly started one-shotting the tank, without the disruption of the instance, this was Blizzard's way of addressing a cheese strat to force the players to do the right as designed: https://www.youtube.com/watch?v=7gMm60BXAjU

Re: Elixir/Erlang Hot Swapping Code (2016)

#30

Post by Muromec »

AnotherGoodName wrote: post #26
AlphaWeaver wrote: post #24

Earlier quoted context omitted.

People may "prefer" simply replacing containers, but as some siblings mention, some applications might require more reliability guarantees. Erlang was originally designed for implementing telephony protocols, where interrupted phone calls were not an acceptable side effect of application updates.

FWIW as soon as you start using containers you should be able to handle those containers spinning up/down. Pretty much the whole point of containers. At which point you don’t need to bother with code hot swapping since you already have a mechanism for newer containers to spin up while older ones spin down. The sibling post “that’s how they update without downtime” is super naive. It is absolutely not how they do it.

That's kinda what erlang does, just on a different level. Your docker and your load balancer are both inside your app.
Post reply on HN