Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upgrading Executable on the Fly

nginx.org

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Upgrading Executable on the Fly

#21

Post by wereHamster »

nullify88 wrote: post #13
wereHamster wrote: post #4

Earlier quoted context omitted.

Even with containerized workloads, you still have an ingress, or SPOF (or multiple, when using multicast), and the seamless restart is meant for exactly those processes. Nginx is often used ( https://kubernetes.github.io/ingress-nginx/ ), or when you use AWS, GCS etc they provide such a service for you. Not sure how the cloud providers do it though, maybe combination of low DNS TTL and rolling restart since they ofte…

A container though should be immutable and ideally shouldn't have changes made to it. If the container were to die, it'd revert back to the old version? It looks to me like these seamless upgrades would be an anti pattern to containers. With ingress you'd have a load balancer in front or have it routed in the network layer using BGP.

How do you restart the load balancer though, without dropping traffic?

Re: Upgrading Executable on the Fly

#22

Post by blibble »

eliaspro wrote: post #15
mholt wrote: post #2

We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go

I'm torn on this feature. In the one hand, an application should never be able to replace itself with "random code" to be executed. I want my systems to be immutable. I want my services to be run with the smallest set of privileges required. On the other hand, it encourages "consumer level" users to keep their software up-to-date, even when it wasn't installed from a distribution's repository etc. So I think in gener…

if you can log into the machine and replace the nginx executable you are probably capable of running it too

Re: Upgrading Executable on the Fly

#23

Post by nullify88 »

rootlocus wrote: post #5

How do the two processes listen to the same port?

Once the USR2 signal is received the master process forks, the child process inherits the parents file descriptors including listen(). One process stops accepting connections creating a queue in the kernel. The new process takes over and starts accepting connections.

You can follow the trail by searching for ngx_exec_new_binary in the nginx repo.

Re: Upgrading Executable on the Fly

#24

Post by fragmede »

JesseObrien wrote: post #14
mholt wrote: post #2

We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go

Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/

If an attacker is already running rogue processes on your box, the minor details surrounding SO_REUSEPORT is the least of your worries. An attacker could just restart nginx, and won't care about lost requests.

Re: Upgrading Executable on the Fly

#25

Post by Grollicus »

wereHamster wrote: post #4
bragr wrote: post #3

I've implemented this a few times in a few languages based on exactly what nginx does. It works well, and it is pretty straight forward if you are comfortable with posix style signals, sockets, and daemons. I'm not sure it is super critical in the age of containerized workloads with rolling deploys but at the very least the connection draining is a good pattern to implement to prevent deploy/scaling related error spi…

Even with containerized workloads, you still have an ingress, or SPOF (or multiple, when using multicast), and the seamless restart is meant for exactly those processes. Nginx is often used ( https://kubernetes.github.io/ingress-nginx/ ), or when you use AWS, GCS etc they provide such a service for you. Not sure how the cloud providers do it though, maybe combination of low DNS TTL and rolling restart since they ofte…

If you really want to have no SPOF you'd probably build something like this:

Multihomed IP Loadbalancer Application

By having the same setup running on multiple locations you can replace the load banacers by taking one location offline (stop announcing the corresponding route). Application instances can be replaced by taking the application instance out of the load balancer.

Re: Upgrading Executable on the Fly

#26

Post by krab »

nullify88 wrote: post #13
wereHamster wrote: post #4

Earlier quoted context omitted.

Even with containerized workloads, you still have an ingress, or SPOF (or multiple, when using multicast), and the seamless restart is meant for exactly those processes. Nginx is often used ( https://kubernetes.github.io/ingress-nginx/ ), or when you use AWS, GCS etc they provide such a service for you. Not sure how the cloud providers do it though, maybe combination of low DNS TTL and rolling restart since they ofte…

A container though should be immutable and ideally shouldn't have changes made to it. If the container were to die, it'd revert back to the old version? It looks to me like these seamless upgrades would be an anti pattern to containers. With ingress you'd have a load balancer in front or have it routed in the network layer using BGP.

I think the parent was talking more about the fact that at some point, you have a component that should be available as much as possible. In the case you mention, that would be the load balancer. Being able to upgrade it in place might be easier than other ways.

Re: Upgrading Executable on the Fly

#27

Post by krab »

nullify88 wrote: post #23
rootlocus wrote: post #5

How do the two processes listen to the same port?

Once the USR2 signal is received the master process forks, the child process inherits the parents file descriptors including listen(). One process stops accepting connections creating a queue in the kernel. The new process takes over and starts accepting connections. You can follow the trail by searching for ngx_exec_new_binary in the nginx repo.

Just to add - Nginx normally spawns several worker processes that all process connections to the same port.

Re: Upgrading Executable on the Fly

#28

Post by cbb330 »

wereHamster wrote: post #21
nullify88 wrote: post #13

Earlier quoted context omitted.

A container though should be immutable and ideally shouldn't have changes made to it. If the container were to die, it'd revert back to the old version? It looks to me like these seamless upgrades would be an anti pattern to containers. With ingress you'd have a load balancer in front or have it routed in the network layer using BGP.

How do you restart the load balancer though, without dropping traffic?

two nginx load balancers, reroute to the secondary via dns, restart primary

Re: Upgrading Executable on the Fly

#29

Post by krab »

majke wrote: post #9

Just a shout out: it's super hard to do it for UDP / QUIC / H3. Beware. (but I don't think nginx supports h3 out of the box yet)

Why so? I thought UDP was stateless, making that process even easier. But I never implemented it.
Post reply on HN