Weekend deployments are for chumps

Post by jacques_chester » Tue, Mar 08, 2011, 11:39 PM UTC

gecko wrote: post #27

viraptor wrote: post #15

I can't believe in how misguided this post is... 1. They provide a service to people around the world, yet they don't ensure that someone is available as an emergency contact on Sunday evening, when the first post-deployment usage happens. 2. They don't have a universal list of "this breaks, contact that guy". 3. They don't have a known instant rollback procedure for a release. 4. They don't have cross-component inte…

EDIT : I posted a rundown of our deployment process, including where and how tests happen, and why they failed to catch this bug, at http://news.ycombinator.com/item?id=2301680 . While I'm sure there's a lot of stuff we could improve, the situation's not exactly as you describe. Responding to a few contacts: 1 & 2. We do have a list of "if this breaks, contact this guy." What we don't have (in response to your first…

> What we don't have (in response to your first point) is a demand that those people be available Sunday night.

Boring, un-agile places have concepts like 24/7 rosters of operations staff and the ability to rotate "on call" duty amongst developers.

However I agree with your conclusion that performing irreversible rollouts are best achieved during (your) daylight hours.

> The real problem here, which has been fixed, is that Kiln should not attempt to hide a problem communicating with FogBugz.

Question: why wasn't an alert raised immediately?

Post by vacri » Wed, Mar 09, 2011, 5:48 AM UTC

zeruch wrote: post #40

Users seem more comfortable with predictable maintenance than arbitrary outages. Weekend deploys are just bad all around. When I began in my current role (managing QA/DBAs and app deploys) one of the first things I killed was the late Friday/weekend deploys. They are spirit-crushing and if they go south, they usually go south in a terminal-velocity nose dive. We set up early Fridays for maintenance, to give us enough…

I seem to accrete job roles, and one time I was so far behind on my testing that I went to the boss to ask for a Wed-Sunday work week so I could get some actual work done without interruptions.

The first fortnight was great, got through a lot of backlog on sat/sun.

The second fortnight sucked as I was blocked on stupidly trivial issues. That ended the experiment.

I guess the moral of the story is to pick and choose your 'out of hours' work wisely.

Re: Weekend deployments are for chumps

Re: Weekend deployments are for chumps