Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Upgrading Executable on the Fly

nginx.org

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Upgrading Executable on the Fly

#51

Post by tyingq »

zimbatm wrote: post #50
tyingq wrote: post #40

Earlier quoted context omitted.

It's kind of fun to watch things go out of fashion and back in. We used to use inetd, mostly because memory was expensive, so it could spawn a service only when a request came in, then the spawned process would exit and give the memory back to the os. Then someone decided tcpd should sit between inetd and servers, for security and logging. Then, every service just ran as it's own daemon. Now I'm occasionally seeing p…

It's the same but a bit different. Inetd is listening to a port and then for each new connection, spawning a new process, binding stdin/stout to the socket pair. The main issue was that it could lead to system resource exhaustion pretty easily if too many connections were being opened and there were no good ways to control that. With systemd, the listening socket is only bound by systemd and passed to the service. Th…

Sure, though systemd does support the inetd style setup also.

Re: Upgrading Executable on the Fly

#52

Post by riyadparvez »

ctrlrsf wrote: post #11
rootlocus wrote: post #5

How do the two processes listen to the same port?

Using socket option SO_REUSEPORT allows multiple processes to bind to same port.

Is there any restrictions on this option? Eg only children of the same parent process are allowed to bind to same port. Otherwise how does the packet distribution work? And how does the response from that port work?

Re: Upgrading Executable on the Fly

#53

Post by tyingq »

cmeacham98 wrote: post #49
tyingq wrote: post #19

Earlier quoted context omitted.

>it can open up that port to hijacking by rogue processes That seems relevant if the process is using a non-privileged port that's >= 1024. If we're talking about privileged ports (<= 1023), though, only another root process could hijack that, and those can already hijack you many other ways.

What about processes that aren't root but hold CAP_NET_BIND_SERVICE?

Sure, should have mentioned that, and perhaps namespaces too.

Re: Upgrading Executable on the Fly

#54

Post by zimbatm »

mholt wrote: post #39
zimbatm wrote: post #37

Earlier quoted context omitted.

If Caddy were to support systemd socket activation, this self-restart dance is not necessary as the parent process (systemd) is holding the socket for you. And for other systems, they can use https://github.com/zimbatm/socketmaster instead. I believe this to be more elegant and robust than the nginx approach as there is no PID re-parenting issues. But I suspect that most Caddy deployments are done via docker, and tha…

Good point, and I'm not sure which deployment method is more popular. In general I am personally not a fan of Docker due to added complexities (often unnecessary for static binaries like Caddy) and technical limitations such as this. All my Caddy deployments use systemd (which I don't love either, sigh).

Technically, it should be possible to pass a file descriptor to a docker process, but I haven't seen that before.

I don't like all parts of systemd but I think the socket activation is pretty elegant. The decoupling also allows to bind on port 80 and 443 (because systemd runs as root), and then still have Caddy run as a user process. I think it would unlock nice things for Caddy.

Re: Upgrading Executable on the Fly

#56

Post by tannhaeuser »

jasonjayr wrote: post #34
jabedude wrote: post #30

Seems like a useful feature for a service manager like systemd to have for its managed services. It is already able to perform inetd style socket activation, I imagine this would be a welcome feature

inetd style socket activation (iirc) forks a process for every connection. So, simply replacing the binary on disk will cause all new connections going forward to use the new binary, while existing held connections (with in-memory references to the old binary's inode) will finish the operations. Once they are done and all references to that inode are gone, the blocks referencing the binary will be removed.

CGI (apache's mod_cgi) has supported this since almost the beginning of the web as well. Deploying a new CGI is as simple as replacing the CGI binary.

Re: Upgrading Executable on the Fly

#57

Post by eliaspro »

mholt wrote: post #16
eliaspro wrote: post #15

Earlier quoted context omitted.

I'm torn on this feature. In the one hand, an application should never be able to replace itself with "random code" to be executed. I want my systems to be immutable. I want my services to be run with the smallest set of privileges required. On the other hand, it encourages "consumer level" users to keep their software up-to-date, even when it wasn't installed from a distribution's repository etc. So I think in gener…

> an application should never be able to replace itself with "random code" to be executed. To clarify: it doesn't, nor has it ever worked that way. You have to be the one to do that (or someone with privileges to write to that file on disk). Most production setups don't give Caddy that permission. And you have to trigger the upgrade too.

Ok, this explains a lot and also clearly shows I never used this feature :)

I always assumed it would be the Caddy process itself taking care of downloading the update & replacing the binary, before restarting it.

Re: Upgrading Executable on the Fly

#58

Post by latchkey »

mholt wrote: post #2

We did this for Caddy 1 too [1]. It was really cool. I am not sure how many people used this feature, so I haven't implemented it for Caddy 2 yet, and in the ~two years that Caddy 2 has been released, I've only had the request once. It's a bit tricky/tedious to do properly, but I'm willing to bring it over to Caddy 2 with a sufficient sponsorship. [1]: https://github.com/caddyserver/caddy/blob/v1/upgrade.go

Someone did this for golang. It isn't perfect, but works for some basic use cases... https://github.com/jpillora/overseer

Re: Upgrading Executable on the Fly

#59

Post by xyzzy_plugh »

tyingq wrote: post #20
duskwuff wrote: post #17

Earlier quoted context omitted.

You don't even need that. If the old server process exec()s the new one, it can pass on its file descriptors -- including the listening socket -- when that happens.

You could also be fancy and pass open sockets over a unix domain socket with sendmsg().

This is the best way as it avoids any sort of session/parenting issues which are not always easy to solve portably as a parent.

Re: Upgrading Executable on the Fly

#60

Post by xyzzy_plugh »

wereHamster wrote: post #4
bragr wrote: post #3

I've implemented this a few times in a few languages based on exactly what nginx does. It works well, and it is pretty straight forward if you are comfortable with posix style signals, sockets, and daemons. I'm not sure it is super critical in the age of containerized workloads with rolling deploys but at the very least the connection draining is a good pattern to implement to prevent deploy/scaling related error spi…

Even with containerized workloads, you still have an ingress, or SPOF (or multiple, when using multicast), and the seamless restart is meant for exactly those processes. Nginx is often used ( https://kubernetes.github.io/ingress-nginx/ ), or when you use AWS, GCS etc they provide such a service for you. Not sure how the cloud providers do it though, maybe combination of low DNS TTL and rolling restart since they ofte…

> Not sure how the cloud providers do it though

At least some I am familiar with operate at the packet level and can hand off live "connections" to a peer or hot standby, along with full session state.

Remember that with TCP or anything else, the abstract session is an illusion.

Post reply on HN