I am curious does anyone know why Nginx uses SIGWINCH for this? I know Apache uses WINCH as well which makes me wonder if there was some historical reason a server process wound up using a signal meant for a TTY?
Upgrading Executable on the Fly
41–50 of 72 posts
Re: Upgrading Executable on the Fly
#42Seems like a useful feature for a service manager like systemd to have for its managed services. It is already able to perform inetd style socket activation, I imagine this would be a welcome feature
inetd style socket activation (iirc) forks a process for every connection. So, simply replacing the binary on disk will cause all new connections going forward to use the new binary, while existing held connections (with in-memory references to the old binary's inode) will finish the operations. Once they are done and all references to that inode are gone, the blocks referencing the binary will be removed.
> inetd style socket activation (iirc) forks a process for every connection.
inetd supports both process-per-connection and single process/multiple connections using the "nowait" and "wait" declarations, respectively. The former passes an accept'd socket, the latter passes the listening socket.
Re: Upgrading Executable on the Fly
#43I've been working on something similar in a load balancer I've been writing in Rust. It's still a work in progress.
Basically the parent executes the new binary after it receives a USR1 signal. Once the child is healthy it kills the parent via SIGTERM. The listener socket file descriptor is passed over an environment variable.
https://github.com/monroeclinton/- (this is the proper url, it's called dash)
Re: Upgrading Executable on the Fly
#44I've considered building something like this to allow for us to update customer software while it's serving users.
In my proposals, there would be a simple application-aware http proxy process that we'd maintain and install on all environments. It would handle relaying public traffic to the appropriate final process on an alternate port. There would be a special pause command we could invoke on the proxy that would buy us time to swap the processes out from under the TCP requests. A second resume command would be issued once the process is running and stable. Ideally, the whole deal completes in ~5 seconds. Rapid test rollbacks would be double that. You can do most of the work ahead of time by toggling between an A and B install path for the binaries, with a third common data path maintained in the middle (databases, config, etc)
With the above proposal, the user experience would be a brief delay at time of interaction, but we already have some UX contexts where delays of up to 30 seconds are anticipated. Absolutely no user request would be expected to drop with this approach, even in a rollback scenario. Our product is broad enough that entire sections of it can be a flaming wasteland while other pockets of users are perfectly happy, so keeping the happy users unbroken is key.
Re: Upgrading Executable on the Fly
#45See Envoy Proxy's Hot Restart [0] 0. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overv...
Also HAProxy, they both use UNIX sockets via ancillary messages+SCM_RIGHTS I believe.
https://www.haproxy.com/blog/truly-seamless-reloads-with-hap...
Re: Upgrading Executable on the Fly
#46I've considered building something like this to allow for us to update customer software while it's serving users. In my proposals, there would be a simple application-aware http proxy process that we'd maintain and install on all environments. It would handle relaying public traffic to the appropriate final process on an alternate port. There would be a special pause command we could invoke on the proxy that would b…
https://en.wikipedia.org/wiki/Blue-green_deployment
DNS not required. You can use a load balancer to do the same thing. If you don't want a full second setup, do a rolling restart of application servers instead.
Edit: I forgot... you can do this with containers too.
Re: Upgrading Executable on the Fly
#47Just a shout out: it's super hard to do it for UDP / QUIC / H3. Beware. (but I don't think nginx supports h3 out of the box yet)
How are quick/http3 servers usually upgraded? As you say, it seems tricky.
Re: Upgrading Executable on the Fly
#48Earlier quoted context omitted.
A container though should be immutable and ideally shouldn't have changes made to it. If the container were to die, it'd revert back to the old version? It looks to me like these seamless upgrades would be an anti pattern to containers. With ingress you'd have a load balancer in front or have it routed in the network layer using BGP.
How do you restart the load balancer though, without dropping traffic?
2 LBs and a VIP/DNS switch.
Re: Upgrading Executable on the Fly
#49Earlier quoted context omitted.
Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/
>it can open up that port to hijacking by rogue processes That seems relevant if the process is using a non-privileged port that's >= 1024. If we're talking about privileged ports (<= 1023), though, only another root process could hijack that, and those can already hijack you many other ways.
What about processes that aren't root but hold CAP_NET_BIND_SERVICE?
Re: Upgrading Executable on the Fly
#50Earlier quoted context omitted.
If Caddy were to support systemd socket activation, this self-restart dance is not necessary as the parent process (systemd) is holding the socket for you. And for other systems, they can use https://github.com/zimbatm/socketmaster instead. I believe this to be more elegant and robust than the nginx approach as there is no PID re-parenting issues. But I suspect that most Caddy deployments are done via docker, and tha…
It's kind of fun to watch things go out of fashion and back in. We used to use inetd, mostly because memory was expensive, so it could spawn a service only when a request came in, then the spawned process would exit and give the memory back to the os. Then someone decided tcpd should sit between inetd and servers, for security and logging. Then, every service just ran as it's own daemon. Now I'm occasionally seeing p…
It's the same but a bit different.
Inetd is listening to a port and then for each new connection, spawning a new process, binding stdin/stout to the socket pair. The main issue was that it could lead to system resource exhaustion pretty easily if too many connections were being opened and there were no good ways to control that.
With systemd, the listening socket is only bound by systemd and passed to the service. The service itself is responsible for accepting and handling the new connections. So it has the control on the rate of new connections, and can also more easily share memory. The main advantage of that approach is that as soon as systemd binds the socket, new connections won't be rejected by the system and will be on hold until the service accept() them. So no connection gets dropped, even during a restart. The service itself is still responsible for gracefully shutting down existing connections on SIGTERM.