Race-condition-free deployment with the "symlink replacement" trick
1–10 of 38 posts
Re: Race-condition-free deployment with the "symlink replacement" trick
#2This seems a bit oversimplified. Sure, there are no race conditions if there are no interactions between files, but if there are then swapping the symlink mid-request on requests that are already in progress will cause all sorts of race conditions.
Re: Race-condition-free deployment with the "symlink replacement" trick
#3Rex [1] already supports deployment (and rollbacks) using symlink replacement.
Re: Race-condition-free deployment with the "symlink replacement" trick
#4I'd prefer to just bring the machine out of rotation, redeploy, bring it back.
Re: Race-condition-free deployment with the "symlink replacement" trick
#5This is a good system at face value, but can present other problems. Any code that has a stat cache (I know PHP does, and I'd be surprised if other common languages don't) suddenly doesn't realize that your paths are going to a different place. Because /var/www as your "base" directory is symlinked to /var/www.a, www.a is cached and when you swap your symlink to www.b to deploy the next version, anything relying on that stat cache (include directives, autoloaders, etc) suddenly starts pulling in the wrong version of the file.
Solving this in a way that doesn't require restarting any services and doesn't introduce any more race conditions is nontrivial, although it tends to work pretty reliably so long as you have a front controller and it very rarely changes. Basically it comes down to a symlink change detector by hitting a special (internal) URL after your deploy script which kills the stat cache. If people are interested I can post a more concrete example.
Re: Race-condition-free deployment with the "symlink replacement" trick
#6This seems a bit oversimplified. Sure, there are no race conditions if there are no interactions between files, but if there are then swapping the symlink mid-request on requests that are already in progress will cause all sorts of race conditions.
Not necessarily. On a Linux-based system, you're actually going to be operating on inodes rather than path names, so once you have a handle to a file, changing the path to it isn't going to affect your ability to interact with it. Just make sure that your new directory is set up so that the same request for the same target file ends up requesting the same inode and you shouldn't notice any problems.
Re: Race-condition-free deployment with the "symlink replacement" trick
#7Yeah, mv is atomic but I believe that a crash can still leave your data inconsistent due to write reordering, see:
http://comments.gmane.org/gmane.os.solaris.opensolaris.zfs/2...
I would do a fsync() before switching the symlink to the new dir.
Re: Race-condition-free deployment with the "symlink replacement" trick
#8Yeah, mv is atomic but I believe that a crash can still leave your data inconsistent due to write reordering, see: http://comments.gmane.org/gmane.os.solaris.opensolaris.zfs/2... I would do a fsync() before switching the symlink to the new dir.
yes. that's a good idea if you care about it. the good part about this solution is that even if the newly deployed file tree becomes corrupted, you can simply point the symlink to the old (certainly not corrupt) tree. that is easy and fast error recovery. an fsync in between makes the window very small, so that only the symlink might get broken.
Re: Race-condition-free deployment with the "symlink replacement" trick
#9This is a good system at face value, but can present other problems. Any code that has a stat cache (I know PHP does, and I'd be surprised if other common languages don't) suddenly doesn't realize that your paths are going to a different place. Because /var/www as your "base" directory is symlinked to /var/www.a, www.a is cached and when you swap your symlink to www.b to deploy the next version, anything relying on t…
I work for a pretty big PHP shop. Our opcode cache has some issues with this, but our stat-cache is ok. Inside the code we unpack the symlink, so internally we are pointing to files like ~/releases//codebase/HTML/index.php instead of ~/codebase/HTML/index.php
Because of the opcode issues we still do a rolling restart of the apaches but we don't see them a lot.
We use `rename` instead of `mv`. Don't know why :)
edit: this also handles the problem of suddenly referencing different files mid-request. Because you don't
Re: Race-condition-free deployment with the "symlink replacement" trick
#10Ok, one problem solved. Now what's left is - schema change, making sure ongoing process flows can automatically migrate from the previous version to the new one, resources referenced from the previous version are still valid, nothing tries to read files with the code via the link (it can change mid-request).
I got the strange feeling from that article as if changing the code files was the hardest thing about yes upgrade.