Weekend deployments are for chumps • Hacker News Board

Re: Weekend deployments are for chumps

#31

Post by tjarratt » Tue, Mar 08, 2011, 5:35 PM UTC

dpritchett wrote: post #10

swombat wrote: post #8

Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...

Maybe in the future we can all be IMVU: Back to the deploy process, nine minutes have elapsed and a commit has been greenlit for the website. The programmer runs the imvu_push script. The code is rsync’d out to the hundreds of machines in our cluster. Load average, cpu usage, php errors and dies and more are sampled by the push script, as a basis line. A symlink is switched on a small subset of the machines throwing…

Tim Fitz' blog is a great source of continuous deployment done right and finding useful information in a sea of chaos.

I remember talking to him before and after he wrote some of these blog posts and it was fascinating seeing how his attitude regarding failure changed.

Re: Weekend deployments are for chumps

#32

Post by mnutt » Tue, Mar 08, 2011, 5:59 PM UTC

gecko wrote: post #28

swombat wrote: post #8

Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...

The releases are both automated (except for one component, as noted, which we are now automating), and are fully vetted. Here is the old release process: 1. Monday morning, the version to be used for the next release is automatically built for the QA team, who begins running their test suites on it and doing soft checks. 2. By no later than Wednesday, the new version is leaked to testing an alpha accounts on Fog Cree…

Perhaps have Kiln send notifications on the failure conditions even if it doesn't throw an error? Better a few false positives than no indication at all.

Re: Weekend deployments are for chumps

#33

Post by tghw » Tue, Mar 08, 2011, 6:02 PM UTC

viraptor wrote: post #15

I can't believe in how misguided this post is... 1. They provide a service to people around the world, yet they don't ensure that someone is available as an emergency contact on Sunday evening, when the first post-deployment usage happens. 2. They don't have a universal list of "this breaks, contact that guy". 3. They don't have a known instant rollback procedure for a release. 4. They don't have cross-component inte…

To clarify a few things:

1. Tim, our sysadmin, was the emergency contact and was on top of things ASAP, but didn't have the particular knowledge to fix it himself. That required a developer.

2. He called me first, since the problem appeared to be in Kiln. I'm a Kiln dev, I can and have fixed things on a Sunday evening after a deploy. I missed his first call, but got back to him within 3 minutes (the 30 minutes mentioned in the article was Ben's guess). I started diagnosing the problem and realized it wasn't Kiln specifically that was the problem, but something in the communication between the two. That meant we needed a FogBugz dev, which we got quickly, and possibly a deploy..

3. That led us to investigate rolling the specific account back to the previous version. 98 percent of our updates are reversible, but as Ben mentioned, this particular release included not one, but two irreversible database migrations, and since the upgrade step had run successfully, going back would not be an option.

4. All tests passed (both automated and manual). Ben has updated the article to make it clear that all but one API call between Kiln and FogBugz was working, and the one call that was broken, the one that lead to this crash, is called very infrequently (on the order of months for some accounts). Yes, integration tests should have and will cover that one API call, but missing one corner case is very different than not doing integration testing.

5. Given the situation we were in, the problem will always be solved more quickly when we're in the office than when we're at home. We take every possible precaution to avoid outages, but they will still happen, and moving to mid-week deploys is just another precaution to decrease the impact of these outages if and when they occur in the future.

So in short, yes, this is the very definition of real software, and we take this very seriously. Your bullet list of armchair quarterback suggestions grossly oversimplifies the situation. The goal is to have problems affect fewer users, which is directly affected by our response time.

Re: Weekend deployments are for chumps

#34

Post by tedunangst » Tue, Mar 08, 2011, 6:02 PM UTC

peterwwillis wrote: post #9

gecko wrote: post #7

Earlier quoted context omitted.

Probably? I'm welcome to be schooled here. 90% of the time, we can roll back instantly, because there were no database changes. 5% of the time, we can roll back with slightly more pain, because the database migrations were reversible. In this case, the database migration was not reversible. If we'd noticed immediately, we could still have just activated snapshots, but we didn't notice until 20 hours later. What do ot…

Wait. What blew up that it took someone 20 hours to realize? The first thing you take from that is, don't do anything without double-checking your change to make sure it worked. In terms of rollback, just don't do anything which isn't reversible. Taking chances with your changes is taking chances with your business. If you don't know how to rollback whatever you're doing, ask someone who does (there is always a way t…

"The failed API call turns out to be one that’s trivially cached for a very long time, and so is one that Kiln would allow to fail without actually dying."

Re: Weekend deployments are for chumps

#35

Post by nollidge » Tue, Mar 08, 2011, 6:05 PM UTC

adamzochowski wrote: post #6

I was taught that Thursdays are best for deployment because you got Friday to fix stupid things, and then weekend to fix the terrible things. By Monday all is working anyways. And best of all, Friday people are generally happy (it is last day of the week), respectively on Monday expect grumpy users.

> Friday people are generally happy

Not if they're trying to get that last thing done before they can go home. :)

Re: Weekend deployments are for chumps

#36

Post by kamens » Tue, Mar 08, 2011, 6:09 PM UTC

swombat wrote: post #8

Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...

It's stunning how easy it is to spot a specific lack of "automated production monitoring" after something fails. Hey idiot, you should've been testing that thing!

I've seen all of Fog Creek's automated production monitoring courtesy of their sysadmins and devs as it was months ago, and it was very solid. I'm sure it's only gotten better.

This is a case of a specific deployment failure slipping through the cracks and being honestly explained, apologized for, and rectified. I'm obviously biased due to my history (and probably-justified guilt for this particular failure), but shotgun criticism about formal release procedures is very misguided.

Re: Weekend deployments are for chumps

#37

Post by aprrrr » Tue, Mar 08, 2011, 6:11 PM UTC

dpritchett wrote: post #10

swombat wrote: post #8

Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...

Maybe in the future we can all be IMVU: Back to the deploy process, nine minutes have elapsed and a commit has been greenlit for the website. The programmer runs the imvu_push script. The code is rsync’d out to the hundreds of machines in our cluster. Load average, cpu usage, php errors and dies and more are sampled by the push script, as a basis line. A symlink is switched on a small subset of the machines throwing…

In fairness, that's a description of what a routine and successful build "should" go like. I bet if IMVU were to post a blow-by-blow account of their hairiest deployment screwup ever, it would be a good bit more colorful than that.

There are some headscratchers in the description of the Fogbugz problem, but kudos to them for explaining how and why things broke.

Re: Weekend deployments are for chumps

#38

Post by bluesnowmonkey » Tue, Mar 08, 2011, 7:47 PM UTC

gecko wrote: post #28

swombat wrote: post #8

Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...

The releases are both automated (except for one component, as noted, which we are now automating), and are fully vetted. Here is the old release process: 1. Monday morning, the version to be used for the next release is automatically built for the QA team, who begins running their test suites on it and doing soft checks. 2. By no later than Wednesday, the new version is leaked to testing an alpha accounts on Fog Cree…

Sincere question: how do you leak irreversible schema changes to a subset of accounts? Isn't the point of the leak that you're not confident and might need to reverse it? Or are you willing to let those accounts get hosed?

Re: Weekend deployments are for chumps

#39

Post by tedunangst » Tue, Mar 08, 2011, 8:22 PM UTC

bluesnowmonkey wrote: post #38

gecko wrote: post #28

Earlier quoted context omitted.

The releases are both automated (except for one component, as noted, which we are now automating), and are fully vetted. Here is the old release process: 1. Monday morning, the version to be used for the next release is automatically built for the QA team, who begins running their test suites on it and doing soft checks. 2. By no later than Wednesday, the new version is leaked to testing an alpha accounts on Fog Cree…

Sincere question: how do you leak irreversible schema changes to a subset of accounts? Isn't the point of the leak that you're not confident and might need to reverse it? Or are you willing to let those accounts get hosed?

Fix it by hand. If it's ten accounts, that's pretty easy. If it's ten thousand, more of a problem.

When you read irreversible, think "very difficult to reverse and not worth the cost of writing and validating code we don't ever expect to run."

Re: Weekend deployments are for chumps

#40

Post by zeruch » Tue, Mar 08, 2011, 9:22 PM UTC

Users seem more comfortable with predictable maintenance than arbitrary outages. Weekend deploys are just bad all around.

When I began in my current role (managing QA/DBAs and app deploys) one of the first things I killed was the late Friday/weekend deploys. They are spirit-crushing and if they go south, they usually go south in a terminal-velocity nose dive.

We set up early Fridays for maintenance, to give us enough time in case something goes south. Aggressive Change Control Requests means the people impacted get a heads up (including Account Managers, who in turn inform clients) if there are any user-facing impacts, and we avoid trying to pack too much in at once.

Having QA, Engineering and the SOC team on hand is...helpful. Maybe its paranoid, but its been very solid so far. When things have gone south, I think the events -since everyone is "on deck" have actually helped build some cameraderie in the teams themselves.