Weekend deployments are for chumps
blog.bitquabit.com
Weekend deployments are for chumps
1–10 of 42 posts
Re: Weekend deployments are for chumps
#2I work for web dev agencies and it surprises me just how often they launch on a Friday afternoon despite how every single developer pleads with them that its an absolutely awful idea.
Golden rule, Never launch on a Friday.
Personally i've found it easy to persuade clients to do this once you say it'll cost an extra ten grand just for the privilege of a friday launch.
Re: Weekend deployments are for chumps
#3I agree that weekend deploys are a shitty idea, but isn't the real issue here not being able to roll back?
Re: Weekend deployments are for chumps
#4I feel for you. On the plus side, process improvements to prevent it from happening next time are exactly how you should respond to things like this.
One which has saved my bacon numerous times is investing a few hours into tweaking monitoring and alert systems. I hear PagerDuty exists to help with this. I use a bunch of scripts and bubblegum, and even that caught 10 of the last 12 big problems. Queuing systems dying has hosed me many times over the years, for example, and a borked deploy which causes that would have my phone ringing before I got my laptop closed.
Re: Weekend deployments are for chumps
#5I'm lucky to run a system that is small enough that an entire deploy consists of around 2 seconds of downtime for the server to restart and start the new instance of the application.
We deploy new versions side by side and then then the webserver points at the new application on restart.
Only time it takes any longer is when there are sweeping database changes(schedule the downtime, inspect snapshots incase of issues, etc.)
Re: Weekend deployments are for chumps
#6I was taught that Thursdays are best for deployment because you got Friday to fix stupid things, and then weekend to fix the terrible things. By Monday all is working anyways.
And best of all, Friday people are generally happy (it is last day of the week), respectively on Monday expect grumpy users.
Re: Weekend deployments are for chumps
#7I agree that weekend deploys are a shitty idea, but isn't the real issue here not being able to roll back?
Probably? I'm welcome to be schooled here. 90% of the time, we can roll back instantly, because there were no database changes. 5% of the time, we can roll back with slightly more pain, because the database migrations were reversible. In this case, the database migration was not reversible. If we'd noticed immediately, we could still have just activated snapshots, but we didn't notice until 20 hours later. What do others do in this situation?
Re: Weekend deployments are for chumps
#8Wait, what?
You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures?
I think your problem is more than just weekend deployments!
My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...
Re: Weekend deployments are for chumps
#9I agree that weekend deploys are a shitty idea, but isn't the real issue here not being able to roll back?
Probably? I'm welcome to be schooled here. 90% of the time, we can roll back instantly, because there were no database changes. 5% of the time, we can roll back with slightly more pain, because the database migrations were reversible. In this case, the database migration was not reversible. If we'd noticed immediately, we could still have just activated snapshots, but we didn't notice until 20 hours later. What do ot…
Wait. What blew up that it took someone 20 hours to realize? The first thing you take from that is, don't do anything without double-checking your change to make sure it worked.
In terms of rollback, just don't do anything which isn't reversible. Taking chances with your changes is taking chances with your business. If you don't know how to rollback whatever you're doing, ask someone who does (there is always a way to roll back or add redundancy).
Re: Weekend deployments are for chumps
#10Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...
Maybe in the future we can all be IMVU:
Back to the deploy process, nine minutes have elapsed and a commit has been greenlit for the website. The programmer runs the imvu_push script. The code is rsync’d out to the hundreds of machines in our cluster. Load average, cpu usage, php errors and dies and more are sampled by the push script, as a basis line. A symlink is switched on a small subset of the machines throwing the code live to its first few customers. A minute later the push script again samples data across the cluster and if there has been a statistically significant regression then the revision is automatically rolled back. If not, then it gets pushed to 100% of the cluster and monitored in the same way for another five minutes. The code is now live and fully pushed. This whole process is simple enough that it’s implemented by a handfull of shell scripts.
http://timothyfitz.wordpress.com/2009/02/10/continuous-deplo...