How do the two processes listen to the same port?
Using socket option SO_REUSEPORT allows multiple processes to bind to same port.
Upgrading Executable on the Fly
11–20 of 72 posts
Re: Upgrading Executable on the Fly
#12Re: Upgrading Executable on the Fly
#13I've implemented this a few times in a few languages based on exactly what nginx does. It works well, and it is pretty straight forward if you are comfortable with posix style signals, sockets, and daemons. I'm not sure it is super critical in the age of containerized workloads with rolling deploys but at the very least the connection draining is a good pattern to implement to prevent deploy/scaling related error spi…
Even with containerized workloads, you still have an ingress, or SPOF (or multiple, when using multicast), and the seamless restart is meant for exactly those processes. Nginx is often used ( https://kubernetes.github.io/ingress-nginx/ ), or when you use AWS, GCS etc they provide such a service for you. Not sure how the cloud providers do it though, maybe combination of low DNS TTL and rolling restart since they ofte…
A container though should be immutable and ideally shouldn't have changes made to it. If the container were to die, it'd revert back to the old version? It looks to me like these seamless upgrades would be an anti pattern to containers.
With ingress you'd have a load balancer in front or have it routed in the network layer using BGP.
Re: Upgrading Executable on the Fly
#14We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on?
Re: Upgrading Executable on the Fly
#15We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
I'm torn on this feature.
In the one hand, an application should never be able to replace itself with "random code" to be executed. I want my systems to be immutable. I want my services to be run with the smallest set of privileges required.
On the other hand, it encourages "consumer level" users to keep their software up-to-date, even when it wasn't installed from a distribution's repository etc.
So I think in general it's a good feature to have, as advanced users/distributions will restrict what a service/process is able to to anyways and won't have any downsides of not using this feature.
It should be optional, that's all!
Re: Upgrading Executable on the Fly
#16We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
I'm torn on this feature. In the one hand, an application should never be able to replace itself with "random code" to be executed. I want my systems to be immutable. I want my services to be run with the smallest set of privileges required. On the other hand, it encourages "consumer level" users to keep their software up-to-date, even when it wasn't installed from a distribution's repository etc. So I think in gener…
> an application should never be able to replace itself with "random code" to be executed.
To clarify: it doesn't, nor has it ever worked that way. You have to be the one to do that (or someone with privileges to write to that file on disk). Most production setups don't give Caddy that permission. And you have to trigger the upgrade too.
Re: Upgrading Executable on the Fly
#17We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/
You don't even need that. If the old server process exec()s the new one, it can pass on its file descriptors -- including the listening socket -- when that happens.
Re: Upgrading Executable on the Fly
#18How do the two processes listen to the same port?
This article on how haproxy uses SO_REUSEPORT goes into some more detail: https://www.haproxy.com/blog/truly-seamless-reloads-with-hap...
Re: Upgrading Executable on the Fly
#19We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go
Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/
>it can open up that port to hijacking by rogue processes
That seems relevant if the process is using a non-privileged port that's >= 1024. If we're talking about privileged ports (<= 1023), though, only another root process could hijack that, and those can already hijack you many other ways.
Re: Upgrading Executable on the Fly
#20Earlier quoted context omitted.
Can you explain any of the technical details around this perchance? I'm super curious. I know that SO_REUSEPORT[1] exists but is that the only little trick to make this work? From what I've read with SO_REUSEPORT it can open up that port to hijacking by rogue processes, so is that fine to rely on? [1] https://lwn.net/Articles/542629/
You don't even need that. If the old server process exec()s the new one, it can pass on its file descriptors -- including the listening socket -- when that happens.
You could also be fancy and pass open sockets over a unix domain socket with sendmsg().