Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Elixir/Erlang Hot Swapping Code (2016)

kennyballou.com

Post reply on HN
Print view View original

61–70 of 119 posts

Re: Elixir/Erlang Hot Swapping Code (2016)

#61

Post by superdisk »

Volundr wrote: post #31

It's worth noting that distillery is deprecated in favor of mix releases, which don't support relups out of the box, and specifically warn against them due to the complexity involved in writing code to support them correctly. It's a cool feature that's no doubt amazing for applications that need it, but it brings a fair amount of complexity vs other deployment strategies.

Yeah, note that this article is from 2016. I distinctly remember during that time that these hot-swap deployments were all the rage in the Elixir community, and then fell out of fashion with time.

Re: Elixir/Erlang Hot Swapping Code (2016)

#62

Post by toast0 »

bitwalker wrote: post #52
Muromec wrote: post #9

Earlier quoted context omitted.

Erlang doesn't have closures, because erlang doesn't have variables. The compiler simply desugars it to partially applied function referenced by it's name (yes, those inline functions in fact have names). If you have something_function, then first inline function used in it will be -something_function/1-fun-0- with zero being the index and captured variable being another argument. Now if you will change the host func…

Erlang absolutely has closures, you are mistaken. What you are referring to are "function captures", which bind a function reference as a value, and there is no environment to close over with those. However, you can define closures which as you'd expect, can close over bindings in the environment in which the closure is defined. The interaction between hot reloads and function captures in general is a bit subtle, par…

> but is force upgraded after two consecutive hot upgrades, as only two versions of a module are allowed to exist at the same time.

Force upgraded is maybe misleading. When a module is loaded for the 3rd time, any processes that still have the first version in their stack are killed. That may result in a supervisor restarting them with new code, if they're supervised.

Re: Elixir/Erlang Hot Swapping Code (2016)

#63

Post by dang »

gnabgib wrote: post #51
dang wrote: post #49

Earlier quoted context omitted.

Where do you see that? I couldn't find it.

It's in the URL :D But yeah, the page doesn't make it clear (and some of the embedded JS has a 2020 date suggesting it's received updates). In the RSS feed too: Wed, 07 Dec 2016 https://kennyballou.com/index.xml

Hidden in plain view! Ok, let's put 2016 above, on the assumption that the edits since then haven't been too major.

Re: Elixir/Erlang Hot Swapping Code (2016)

#64

Post by tomjakubowski »

Thaxll wrote: post #28
diath wrote: post #16

Earlier quoted context omitted.

> in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container How do you think video games like World of Warcraft or Path of Exile deploy restartless hotfixes to millions of concurrent players without killing instances? I don't think it's a matter of "prefer to", it's a matter of "can we completely disrupt the service for users and pote…

Most of those hot fixes are data driven as in database updates. Gameserver just reload the data, the binary itself is not touch. I've never seen a game where they hot reload code inside the gameserver itself, it's usually a downtime or rolling updates.

LPMUDs ran almost entirely on hot reloadable code written in a quirky language called LPC, which later inspired the Pike language.

I believe that only the "driver" code, which handles system calls and hosts the LPC interpreter and is written in C, couldn't be hot reloaded; everything else running in the game could be reloaded without restarting the server.

I'd guess in the modern day, there would be some games where Lua scripts can be hot-reloaded like any other data, from a database or object store.

Re: Elixir/Erlang Hot Swapping Code (2016)

#65

Post by Muromec »

bitwalker wrote: post #52
Muromec wrote: post #9

Earlier quoted context omitted.

Erlang doesn't have closures, because erlang doesn't have variables. The compiler simply desugars it to partially applied function referenced by it's name (yes, those inline functions in fact have names). If you have something_function, then first inline function used in it will be -something_function/1-fun-0- with zero being the index and captured variable being another argument. Now if you will change the host func…

Erlang absolutely has closures, you are mistaken. What you are referring to are "function captures", which bind a function reference as a value, and there is no environment to close over with those. However, you can define closures which as you'd expect, can close over bindings in the environment in which the closure is defined. The interaction between hot reloads and function captures in general is a bit subtle, par…

What does is it look like? I was talking about this thing:

   Val = 1, SumFun = fun(X) -> X + Val end, SumFun(2).
It looks like you define arity 1 function that captures Val, while in fact you define arity 2 function and bind 1 as a first argument. Since you can't redefine Val anyway, it's as good as a closure, but technically it doesn't capture the environment.

Maybe I'm mistaken and there is another way to express it?

Re: Elixir/Erlang Hot Swapping Code (2016)

#66

Post by Muromec »

simoncion wrote: post #50
Muromec wrote: post #30

Earlier quoted context omitted.

That's kinda what erlang does, just on a different level. Your docker and your load balancer are both inside your app.

If we to wedge how Erlang does hot code swapping into a container metaphor, then to get what Erlang does, you'd need to have a container per function call. Given that it would be absurdly wasteful to use OS processes in containers to clone Erlang's code reload system, AnotherGoodName might take ten minute to watch Erlang: The Movie to get a better sense of the capabilities of that system. The movie is available from…

>If we to wedge how Erlang does hot code swapping into a container metaphor, then to get what Erlang does, you'd need to have a container per function call.

You have a container that responds to HTPP requests sitting behind a load balancer, then you spawn a new container and tell load balancer to redirect calls to the new one. From the point of view of whoever is calling the load balancer you have hot swapping. You may even separate containers into logical groups and call it microservices architecture. Or you can define a process as something having qualified name and a mailbox and is sending messages to other processes.

Now reasonable people may disagree about what's wasteful, but the market seems to tolerate places where adding a checkbox to a form is a half a year process involving five different departments and the market can't be wrong.

Re: Elixir/Erlang Hot Swapping Code (2016)

#68

Post by deathtrader666 »

hauxir wrote: post #39

At kosmi.io we use elixir hot swapping for every small patch/bugfix on the backend. This allows us to deploy updates multiple times a day with 0 disruption. Allows the clients to remain connected and be none the wiser that there was an update at all. For larger updates we just do hard restarts when in-memory data structures or supervision tree are changed.

Would love to know more how you go about it.

Re: Elixir/Erlang Hot Swapping Code (2016)

#69

Post by toast0 »

anonymousDan wrote: post #8

I'm a distributed setup I imagine there could be cases where you want to atomically hot upgrade multiple VMs at the same time. Is this common in practice and if so are there recommended patterns/techniques for doing it?

I mean, yes, there's cases where you want that. But there's no mechanism for it, because you would have to stop the world, do the load, and then resume.

Even within a single VM, hot loading doesn't stop the world, during the load some schedulers will switch before others. Although there are guarantees that mean when a process runs new code and sends a message to another local process, that process will have the new code available when it reads the message. (It may still be running the old code, depending on how it's called though)

Dealing with multiple versions active is part of life in most distributed systems though. You can architect it away in some systems, but that usually involves having downtime in maintenance windows.

A typical pattern is making progressive updates, where if you want to change a request, first you deploy a server that can handle old and new requests, then you deploy the client that sends the new request, then you can deploy a server that no longer accepts old requests.

For new replies, if the new reply comes with a new request, that works like above... a client that sent a new request must handle the new reply. Otherwise, update the client to handle either type of reply, then update the server to send the new reply, finally remove handling of the old reply in the clients.

It gets a bit harder if your team dynamics mean one person/group doesn't control both sides... Then you need stats to tell you when all the clients have switched.

Sometimes you do need more of a point in time switch. If it needs to be pretty good, you can just set a config through a dist 'broadcast'. If it needs to be better than that, you can have the servers and clients change behavior after a specific time... but make sure you understand the realities of clock synchronization and think about what to do for requests in flight. If that's not good enough, you can drop or buffer requests for a little bit before your targer time, make sure there are no in progress requests, then resume processing requests with the new version.

Re: Elixir/Erlang Hot Swapping Code (2016)

#70

Post by toast0 »

slt2021 wrote: post #5

hot reload of code is nothing new nowadays, but people use it only locally during development for REPL like development style. in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container

> in actual production, people prefer to operate at the container level + traffic management, and dont touch anything deeper than the container

I mean, this seems to be "best practices" these days, but I certainly don't prefer it. At least the orchestration I use is amazingly slow. And cold loading changes is terrible for long running processes... this makes deployment a major chore.

It's less terrible if you're just doing mostly stateless web stuff, but that's not my world.

In the time it takes to run terraform plan, I could have pushed erlang code to all my machines, loaded it, and (usually) confirm I fixed what I wanted to fix.

Low cost of deploy means you can do more updates which means they can be smaller which makes them easier to review.

Post reply on HN