Isn't this guy reinventing a simplified wheel? We already have tools like Capistrano and Fabric or Rex (which does quite a bit more than just application deployments).
Race-condition-free deployment with the "symlink replacement" trick
11–20 of 38 posts
Re: Race-condition-free deployment with the "symlink replacement" trick
#12This is a good system at face value, but can present other problems. Any code that has a stat cache (I know PHP does, and I'd be surprised if other common languages don't) suddenly doesn't realize that your paths are going to a different place. Because /var/www as your "base" directory is symlinked to /var/www.a, www.a is cached and when you swap your symlink to www.b to deploy the next version, anything relying on t…
I've been using this pattern for almost 2 years, and indeed it presented a problem at first with bytecode cache. We've added a simple post deploy script that flushes APC and it fixed this.
Re: Race-condition-free deployment with the "symlink replacement" trick
#13This seems a bit oversimplified. Sure, there are no race conditions if there are no interactions between files, but if there are then swapping the symlink mid-request on requests that are already in progress will cause all sorts of race conditions.
Not necessarily. On a Linux-based system, you're actually going to be operating on inodes rather than path names, so once you have a handle to a file, changing the path to it isn't going to affect your ability to interact with it. Just make sure that your new directory is set up so that the same request for the same target file ends up requesting the same inode and you shouldn't notice any problems.
But what if you don't have a handle to a file? Say you have index.php, which includes stuff.php. Order of operations:
1. Server opens (old) index.php and begins executing
2. Symlink swap
3. Interpreter gets to the line that includes stuff.php, and then opens the *new* fileRe: Race-condition-free deployment with the "symlink replacement" trick
#14This seems a bit oversimplified. Sure, there are no race conditions if there are no interactions between files, but if there are then swapping the symlink mid-request on requests that are already in progress will cause all sorts of race conditions.
Not necessarily. On a Linux-based system, you're actually going to be operating on inodes rather than path names, so once you have a handle to a file, changing the path to it isn't going to affect your ability to interact with it. Just make sure that your new directory is set up so that the same request for the same target file ends up requesting the same inode and you shouldn't notice any problems.
"Just make sure that your new directory is set up so that the same request for the same target file ends up requesting the same inode"
If that is your goal, why do the mv at all?
...until your program needs to work with more than one file and the two are related in some way or opens 'the same file' twice.
For example, your compiler could fetch file X from the 'old' directory and file Y from the 'new' one, or your web server could log that it fetched file Z (which does not exist in the 'new' directory) to a log file in the 'new' directory.
It may be possible to mitigate this by requiring everybody to access all fils true a directory you opened atomically for them, but even if it is: good luck enforcing that rule, especially when using third party libraries.
Re: Race-condition-free deployment with the "symlink replacement" trick
#15I wonder what problem this is really solving. I mean, delete+create in a script will happen pretty fast after each other, so the moment of inconsistency is really very short. If this is an actual problem for you, chances are that your setup is rather large and you have multiple nodes behind a load balancer. In that case, you have bigger issues, such as making sure the individual nodes are updated at the same time. Usually this would be solved by taking them out of rotation while updating, in which case the atomic symlink switch becomes moot.
Re: Race-condition-free deployment with the "symlink replacement" trick
#16I wonder what problem this is really solving. I mean, delete+create in a script will happen pretty fast after each other, so the moment of inconsistency is really very short. If this is an actual problem for you, chances are that your setup is rather large and you have multiple nodes behind a load balancer. In that case, you have bigger issues, such as making sure the individual nodes are updated at the same time. Us…
The problem is that the issues created by out-of-sync code that happens to get loaded in the wrong order can be an absolute nightmare to debug. You'll have transient, unreproducible, potentially data-destroying bugs which vary with each release. Some releases you might get lucky, some not. If you don't think about atomicity of the deployment, you can chase your tail for days trying to figure out what went wrong.
That being said, this strikes me as more of a pain under the traditional PHP model, where reloading code from disc per request is normal, than for something like Rails which loads everything into memory once at launch.
Re: Race-condition-free deployment with the "symlink replacement" trick
#17This system ensures the server is always in a consistent state, but client race conditions are still possible if the "old" index.html references an asset that isn't available after the deployment has occurred. Is there any good way of dealing with this? (I just ignore it...)
Re: Race-condition-free deployment with the "symlink replacement" trick
#18Earlier quoted context omitted.
Not necessarily. On a Linux-based system, you're actually going to be operating on inodes rather than path names, so once you have a handle to a file, changing the path to it isn't going to affect your ability to interact with it. Just make sure that your new directory is set up so that the same request for the same target file ends up requesting the same inode and you shouldn't notice any problems.
But what if you don't have a handle to a file? Say you have index.php, which includes stuff.php. Order of operations: 1. Server opens (old) index.php and begins executing 2. Symlink swap 3. Interpreter gets to the line that includes stuff.php, and then opens the *new* file
That's precisely why the classical PHP model is kinda stuffed under this situation.
If you want to stick with the traditional disc-read-per-request model, I'd be interested to see if something like a blue/green code deployment could work. You'd have two separate html roots - say, with /var/www/blue being current. You deploy your new code to /var/www/green, update httpd.conf to point there, then SIGHUP apache. The next deployment would switch back from /var/www/green to /var/www/blue. That way every request sees a consistent deployment from start to end.
Re: Race-condition-free deployment with the "symlink replacement" trick
#19Isn't this guy reinventing a simplified wheel? We already have tools like Capistrano and Fabric or Rex (which does quite a bit more than just application deployments).
Thats like saying why understand how writing a file to disk works when we have things like notepad and emacs. What do you think these tools do under the hood?
Re: Race-condition-free deployment with the "symlink replacement" trick
#20I wonder what problem this is really solving. I mean, delete+create in a script will happen pretty fast after each other, so the moment of inconsistency is really very short. If this is an actual problem for you, chances are that your setup is rather large and you have multiple nodes behind a load balancer. In that case, you have bigger issues, such as making sure the individual nodes are updated at the same time. Us…
The problem is that the issues created by out-of-sync code that happens to get loaded in the wrong order can be an absolute nightmare to debug. You'll have transient, unreproducible, potentially data-destroying bugs which vary with each release. Some releases you might get lucky, some not. If you don't think about atomicity of the deployment, you can chase your tail for days trying to figure out what went wrong. That…
With APC (and stat disabled) PHP behaves in exactly the same way. Bytecode is kept in memory between requests and you can then safely push a new version of your directory tree. All that is required is to flush the cache to have the new version go live.