So if I understood correctly, would it be like this
cp new/nginx /path/to/nginx kill -SIGUSR2
That does sound pretty neat if you're not running nginx in a container. I wonder if they've built a Windows equivalent for that.
Upgrading Executable on the Fly
31–40 of 72 posts
Re: Upgrading Executable on the Fly
#32Earlier quoted context omitted.
A container though should be immutable and ideally shouldn't have changes made to it. If the container were to die, it'd revert back to the old version? It looks to me like these seamless upgrades would be an anti pattern to containers. With ingress you'd have a load balancer in front or have it routed in the network layer using BGP.
How do you restart the load balancer though, without dropping traffic?
You would need more than one to do a rolling restart. Alternatively to do it with one instance of a software load balancer is a bit more work, spin another instance up and update DNS. Wait for traffic to the old one to die as TTLs expire, then decommission.
But I agree it isn't as easy as a in place upgrade.
Re: Upgrading Executable on the Fly
#33Seems like a useful feature for a service manager like systemd to have for its managed services. It is already able to perform inetd style socket activation, I imagine this would be a welcome feature
This is already possible. You can configure whether you want inetd style socket activation (where systemd calls accept() and passes you the client socket)), or just systemd listening to the socket (where systemd passes you the listen socket and your binary calls accept()).
https://www.freedesktop.org/software/systemd/man/systemd.soc...
Re: Upgrading Executable on the Fly
#34Seems like a useful feature for a service manager like systemd to have for its managed services. It is already able to perform inetd style socket activation, I imagine this would be a welcome feature
inetd style socket activation (iirc) forks a process for every connection.
So, simply replacing the binary on disk will cause all new connections going forward to use the new binary, while existing held connections (with in-memory references to the old binary's inode) will finish the operations. Once they are done and all references to that inode are gone, the blocks referencing the binary will be removed.
Re: Upgrading Executable on the Fly
#35Just a shout out: it's super hard to do it for UDP / QUIC / H3. Beware. (but I don't think nginx supports h3 out of the box yet)
Why so? I thought UDP was stateless, making that process even easier. But I never implemented it.
UDP itself is stateless, but QUIC itself is stateful. Without knowing the background I would assume the issue to be that the incoming UDP packets will be routed to the new process after the reload and that new process is not aware of the existing QUIC connections, because the state resides in the old process. Thus it is not able to decrypt the packets for example.
Re: Upgrading Executable on the Fly
#36Earlier quoted context omitted.
Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/
You don't even need that. If the old server process exec()s the new one, it can pass on its file descriptors -- including the listening socket -- when that happens.
Yep, we don't use SO_REUSEPORT. We just pass it from the old process to the new one.
Re: Upgrading Executable on the Fly
#37We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
If Caddy were to support systemd socket activation, this self-restart dance is not necessary as the parent process (systemd) is holding the socket for you. And for other systems, they can use https://github.com/zimbatm/socketmaster instead. I believe this to be more elegant and robust than the nginx approach as there is no PID re-parenting issues.
But I suspect that most Caddy deployments are done via docker, and that requires a whole container restart anyways.
Re: Upgrading Executable on the Fly
#38Earlier quoted context omitted.
Once the USR2 signal is received the master process forks, the child process inherits the parents file descriptors including listen(). One process stops accepting connections creating a queue in the kernel. The new process takes over and starts accepting connections. You can follow the trail by searching for ngx_exec_new_binary in the nginx repo.
Just to add - Nginx normally spawns several worker processes that all process connections to the same port.
Correct but to clarify, only the master process binds to the ports. The master process creates socketpairs to the workers for interprocess communication. The workers accept connections over the shared socket.
https://www.nginx.com/blog/socket-sharding-nginx-release-1-9...
Page also has an example of how SO_REUSEPORT effects flow.
Re: Upgrading Executable on the Fly
#39We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
If Caddy were to support systemd socket activation, this self-restart dance is not necessary as the parent process (systemd) is holding the socket for you. And for other systems, they can use https://github.com/zimbatm/socketmaster instead. I believe this to be more elegant and robust than the nginx approach as there is no PID re-parenting issues. But I suspect that most Caddy deployments are done via docker, and tha…
Good point, and I'm not sure which deployment method is more popular.
In general I am personally not a fan of Docker due to added complexities (often unnecessary for static binaries like Caddy) and technical limitations such as this. All my Caddy deployments use systemd (which I don't love either, sigh).
Re: Upgrading Executable on the Fly
#40We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
If Caddy were to support systemd socket activation, this self-restart dance is not necessary as the parent process (systemd) is holding the socket for you. And for other systems, they can use https://github.com/zimbatm/socketmaster instead. I believe this to be more elegant and robust than the nginx approach as there is no PID re-parenting issues. But I suspect that most Caddy deployments are done via docker, and tha…
It's kind of fun to watch things go out of fashion and back in. We used to use inetd, mostly because memory was expensive, so it could spawn a service only when a request came in, then the spawned process would exit and give the memory back to the os. Then someone decided tcpd should sit between inetd and servers, for security and logging. Then, every service just ran as it's own daemon. Now I'm occasionally seeing posts like this reviving inetd.