Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Race-condition-free deployment with the "symlink replacement" trick

gist.github.com

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Race-condition-free deployment with the "symlink replacement" trick

#31

Post by borlak »

mjs wrote: post #17

This system ensures the server is always in a consistent state, but client race conditions are still possible if the "old" index.html references an asset that isn't available after the deployment has occurred. Is there any good way of dealing with this? (I just ignore it...)

There is a simple way to solve the original problem and the one you mentioned. A load balancer. One that you have a good amount of control of and an API (such as Zeus, F5...)

You basically take the nodes off one at a time, wait for connections to finish, sync over code, then bring it back up. This does make some assumptions about assets -- that they are in a different location, such as a CDN or static server. If you are removing assets, you need to do this at the very end of this node-syncing process, so that any live "old nodes" aren't linking to deleted assets.

As for newly updated assets, you should be doing versioning for those anyway (even this 'symlink trick' fails when multiple application servers are involved and no shared code space).

Re: Race-condition-free deployment with the "symlink replacement" trick

#34

Post by Tobu »

daenney wrote: post #11

Isn't this guy reinventing a simplified wheel? We already have tools like Capistrano and Fabric or Rex (which does quite a bit more than just application deployments).

After my experience of Capistrano and Fabric, I think it's a wheel that needs to be reinvented. Capistrano: hardcodes way too many things, underdocumented. Fabric: reimplements ssh (poorly). Moving from Capistrano to a custom script I ended up with less code and higher maintainability.

Re: Race-condition-free deployment with the "symlink replacement" trick

#35

Post by sirclueless »

j_baker wrote: post #32

Is it just me, or does this seem like a lot of work just to avoid having assets be inconsistent for "some number of milliseconds"?

Especially when browsers are stateless and request things at different times anyways. If you have a lot of requests, you can expect a few of them to get the HTML from one version and the javascript and CSS from another anyways, no matter your strategy.

I still think this is a valuable deployment strategy, just because you can rollback and switch deployed versions easily, which is always useful. And it's certainly better than rsyncing to a live directory, at any rate.

Re: Race-condition-free deployment with the "symlink replacement" trick

#36

Post by standel »

mv is relying on operation "similar" to rename() defined by POSIX which specifies that it should be atomic.

So, the assumption "On Unix, mv is atomic operation" is not true. If your underlying FS is fully POSIX-compliant, mv will be an atomic operation.

I think it's important to stress it because there are some distributed FS that might even try to be POSIX-compliant but which are not guaranteeing atomic rename's and therefore this trick would not work well.

Re: Race-condition-free deployment with the "symlink replacement" trick

#37

Post by Negitivefrags »

peterwwillis wrote: post #26
Negitivefrags wrote: post #22

We use use this technique at Grinding Gear Games for our deployments but here are a few random assorted details about how we are set up. The first is that we find it's a good idea to have your release directories on the server named after tags from your VCS. Each time we want to do a deploy we just make a tag and the deployment script just takes the name of the tag to deploy as it's argument. It's very easy to see wh…

Nice. It's almost rare these days to see a shop manage its deploys like application releases. Do you do staged production deploys of new code for small groups of users? I found it was beneficial to be able to test a change on a random subset of users so if there's a production-only bug it doesn't hit everyone at once. This also allows you to not have to "stop" the app servers because you're starting up the new versio…

We don't do that on the production realm, but it's kind of because we are a game and patches are a big deal for our community. It's not like most websites where you often don't know when patches are coming or what they changed. We keep full change logs here: http://www.pathofexile.com/forum/view-forum/366

It's worth bearing in mind that we are actually deploying an application that they play on their desktop machines, it's just that our website is tightly integrated with the live realm so they are deployed together in the same deployment system.

What we do have as a game though is the ability to have a separate alpha realm that we can deploy to for testing a release and we have a trusted set of our player base that is allowed access to it.

So here is the list of realms we have:

Testing (Local continuously integrated deploy of trunk. Updated every commit)

Staging1 (Local staging for the next major patch)

Staging2 (Local staged copy of whatever is on production. This is used for when we want to test bugfixes to production)

Alpha (Deploy of the next major patch for some community members to play and test in advance. This is deployed alongside the production realm on the live servers.)

Production

All of that said though, we are adding the ability very soon for the backend to be able to spawn game instance servers for multiple versions of the realm. This would mean that we can deploy a game patch without a restart (assuming the backend didn't change). Old clients would get old game instance servers but as players restart their game client and patch, they will get on new game instance servers.

Re: Race-condition-free deployment with the "symlink replacement" trick

#38

Post by ibotty »

viraptor wrote: post #10

Ok, one problem solved. Now what's left is - schema change, making sure ongoing process flows can automatically migrate from the previous version to the new one, resources referenced from the previous version are still valid, nothing tries to read files with the code via the link (it can change mid-request). I got the strange feeling from that article as if changing the code files was the hardest thing about yes upgr…

you are certainly right. db schema updates are the hard part.

for db schema updates: have a look at sqitch by postgres' david wheeler. it should also support mysql (or will in the future).

Post reply on HN