21–30 of 38 posts
This system ensures the server is always in a consistent state, but client race conditions are still possible if the "old" index.html references an asset that isn't available after the deployment has occurred. Is there any good way of dealing with this? (I just ignore it...)
The first is that we find it's a good idea to have your release directories on the server named after tags from your VCS. Each time we want to do a deploy we just make a tag and the deployment script just takes the name of the tag to deploy as it's argument. It's very easy to see what version is deployed on a server by just looking at the address of the symlink.
The second is that you should use rsync with the --link-dest option. --link-dest allows you to specify a previous directory that rsync can use to create hard links from for files that haven't changed. For example, if you a new version to deploy in a directory called "0.9.10/2" and on the remote server you have "0.9.10/1" currently deployed, you can "rsync 0.9.10/2 server:0.9.10/2 --link-dest 0.9.10/1". What this does is create a new dir tree in /2 with all the files that didn't change from /1 hard linked but with new copies for the files that did. This saves a lot of disk space and it means you can keep versions around on the server for as long as you feel the need to.
As our deployment is ~8GB this is quite important for us. This means that we actually have releases sitting on the server for quite a while back.
The third thing is setting something up so you can have simple versioning of your deployment scripts.
We have a script that drives this whole process called "./realmctl". Deployment is split in to a 4 step process. You find scripts like this in each release dir like this:
./0.9.10/1/prepare (create/upload new release)
./0.9.10/1/stop (stop existing servers)
./0.9.10/1/deploy (change symlinks over to this release)
./0.9.10/1/start (start servers)
Each of the releases contains it's own version of the script. That means if you issue a command like "./realmctl restart --release=0.9.10/2" then the script can find the stop script for the current version then run the deploy and start scripts for the new version. In this way if your deployment process changes between versions then you can still freely move around between versions without needing to worry about the version of your deployment scripts.
The last thing is that it's really nice if your writing something similar for your scripts to have some idea about different parts of your infrastructure so that they can be controlled independently. It's really useful to be able to say something like "./realmctl restart all poe_webserver" (restart webserver processes on all servers) or "./realmctl stop ggg4 poe_instance" (stop the game instance servers on ggg4). Those kind of commands are really useful during an emergency.
This system ensures the server is always in a consistent state, but client race conditions are still possible if the "old" index.html references an asset that isn't available after the deployment has occurred. Is there any good way of dealing with this? (I just ignore it...)
Two phases. Leave possibly referenced assets for a while until use of cached old files has become rare enough, then remove them.
We use use this technique at Grinding Gear Games for our deployments but here are a few random assorted details about how we are set up. The first is that we find it's a good idea to have your release directories on the server named after tags from your VCS. Each time we want to do a deploy we just make a tag and the deployment script just takes the name of the tag to deploy as it's argument. It's very easy to see wh…
Do you do staged production deploys of new code for small groups of users? I found it was beneficial to be able to test a change on a random subset of users so if there's a production-only bug it doesn't hit everyone at once.
This also allows you to not have to "stop" the app servers because you're starting up the new version's instance in parallel with the old. The frontend just passes user-specific requests to the new instance and the old instance keeps chugging along with no downtime. Of course this usually requires no schema changes (unless you have lots of spare infrastructure handy).
This system ensures the server is always in a consistent state, but client race conditions are still possible if the "old" index.html references an asset that isn't available after the deployment has occurred. Is there any good way of dealing with this? (I just ignore it...)
As someone mentioned before, hard-linking old and new deploy files means duplicated content doesn't cost any disk space. Rotate out old deploys past X days, and use strong cache controls to expire the content quickly.
Earlier quoted context omitted.
Not necessarily. On a Linux-based system, you're actually going to be operating on inodes rather than path names, so once you have a handle to a file, changing the path to it isn't going to affect your ability to interact with it. Just make sure that your new directory is set up so that the same request for the same target file ends up requesting the same inode and you shouldn't notice any problems.
But what if you don't have a handle to a file? Say you have index.php, which includes stuff.php. Order of operations: 1. Server opens (old) index.php and begins executing 2. Symlink swap 3. Interpreter gets to the line that includes stuff.php, and then opens the *new* file
I wonder what problem this is really solving. I mean, delete+create in a script will happen pretty fast after each other, so the moment of inconsistency is really very short. If this is an actual problem for you, chances are that your setup is rather large and you have multiple nodes behind a load balancer. In that case, you have bigger issues, such as making sure the individual nodes are updated at the same time. Us…
We handled deployments a different way though. Each release went into a different directory and was made available under a different url, and users got redirected to the newest release when they logged in. Once in a session they stayed with the same version until they logged out. This also allowed us to do limited deployments; we could choose which version each customer group was sent to.
This system ensures the server is always in a consistent state, but client race conditions are still possible if the "old" index.html references an asset that isn't available after the deployment has occurred. Is there any good way of dealing with this? (I just ignore it...)
It seemed to work well, but there are issues with tags you should read up on first.