How We Designed for Performance and Scale
1–10 of 63 posts
Re: How We Designed for Performance and Scale
#2"NGINX’s binary upgrade process achieves the holy grail of high-availability; you can upgrade the software on the fly, without any dropped connections, downtime or interruption in service."
Is this really true? I remember seeing an article[1] recently on using an iptables hack to prevent dropping connections when reloading haproxy. Does nginx actually provide zero-downtime configuration reloads?
[1] https://medium.com/@Drew_Stokes/actual-zero-downtime-with-ha...
Re: How We Designed for Performance and Scale
#3"NGINX’s binary upgrade process achieves the holy grail of high-availability; you can upgrade the software on the fly, without any dropped connections, downtime or interruption in service." Is this really true? I remember seeing an article[1] recently on using an iptables hack to prevent dropping connections when reloading haproxy. Does nginx actually provide zero-downtime configuration reloads? [1] https://medium.co…
If it's the holy grail, then erlang has had it for a couple decades... and whether it actually works in nginx I don't know, but it does work in erlang.
The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.
Re: How We Designed for Performance and Scale
#4Btw there is a new feature in the kernel to help avoid using the accept shared memory mutex -- EPOLLEXCLUSIVE and EPOLLROUNDROBIN
This should round robin accept in the kernel, and not wake up all the epoll listeners.
Re: How We Designed for Performance and Scale
#5"NGINX’s binary upgrade process achieves the holy grail of high-availability; you can upgrade the software on the fly, without any dropped connections, downtime or interruption in service." Is this really true? I remember seeing an article[1] recently on using an iptables hack to prevent dropping connections when reloading haproxy. Does nginx actually provide zero-downtime configuration reloads? [1] https://medium.co…
If it's the holy grail, then erlang has had it for a couple decades... and whether it actually works in nginx I don't know, but it does work in erlang. The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.
"The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code."
That would imply that two separate processes are bound to the same port right? I thought that was not possible.
Re: How We Designed for Performance and Scale
#6Earlier quoted context omitted.
If it's the holy grail, then erlang has had it for a couple decades... and whether it actually works in nginx I don't know, but it does work in erlang. The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.
"The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code." That would imply that two separate processes are bound to the same port right? I thought that was not possible.
Things have changed :)
Re: How We Designed for Performance and Scale
#7Earlier quoted context omitted.
If it's the holy grail, then erlang has had it for a couple decades... and whether it actually works in nginx I don't know, but it does work in erlang. The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.
"The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code." That would imply that two separate processes are bound to the same port right? I thought that was not possible.
No, usually there's an accept process spawning working processes and passing the connection off to it.
Erlang can have two versions of a module present in memory at a time. At the moment of a code upgrade, the processes working with existing connections continue to use the old code while newly accepted connections are passed to the version of the module that was loaded in between connection accepts.
Re: How We Designed for Performance and Scale
#8Earlier quoted context omitted.
If it's the holy grail, then erlang has had it for a couple decades... and whether it actually works in nginx I don't know, but it does work in erlang. The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code.
"The way it's handled is that the process where the previous socket was connected remains in place until it terminates but all new sockets connect to the new code." That would imply that two separate processes are bound to the same port right? I thought that was not possible.
Keep in mind that Erlang processes != System processes. Erlang processes are preemptively scheduled microthreads that are scheduled by the Erlang VM.
Re: How We Designed for Performance and Scale
#9"NGINX’s binary upgrade process achieves the holy grail of high-availability; you can upgrade the software on the fly, without any dropped connections, downtime or interruption in service." Is this really true? I remember seeing an article[1] recently on using an iptables hack to prevent dropping connections when reloading haproxy. Does nginx actually provide zero-downtime configuration reloads? [1] https://medium.co…
Yes, nginx allows you to upgrade the binary and reload the configuration without any downtime: http://wiki.nginx.org/CommandLine#Upgrading_To_a_New_Binary_...
Re: How We Designed for Performance and Scale
#10"NGINX’s binary upgrade process achieves the holy grail of high-availability; you can upgrade the software on the fly, without any dropped connections, downtime or interruption in service." Is this really true? I remember seeing an article[1] recently on using an iptables hack to prevent dropping connections when reloading haproxy. Does nginx actually provide zero-downtime configuration reloads? [1] https://medium.co…
I didn't want to be a negative nancy in the comments for that haproxy article... but that is a ridiculous ugly hack. It's really a lot easier to achieve real and robust zero-downtime upgrades for a simple unix process.
Remember, fork()ed and exec()ed processes inherit file descriptors (except those marked CLOEXEC), including the listen() fd. Pending connections will queue in the kernel until userspace calls accept() on the listening fd.
So one simple model is to stop calling accept(), cleanly/quickly finish up current connections, set an environment variable to tell the future instance that the listening fd X is already open, and exec your own binary again.
A more complicated one is to fork, have the (identical) child just finish the current connections, while the parent execs itself similar to above. (The client connection fds should be marked CLOEXEC in this case.)
With a more complicated service with more moving parts, libraries, threads, getting the above to work out is more complicated. But that's basically how you want to do it.