Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weekend deployments are for chumps

blog.bitquabit.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Weekend deployments are for chumps

#21

Post by brown9-2 »

swombat wrote: post #17
brown9-2 wrote: post #13

Earlier quoted context omitted.

Two better approaches come to mind to resolve this: 2. Full-on, properly managed releases like they do in large IT corporations, such as banks, where a "release" is not something you kick off from home via SSH on a Saturday night, but a properly planned effort that involves critical members of the dev team as well as the QA team being present and ready to both test the production system thoroughly and fix any issues…

It absolutely is, and I'd be surprised to see this kind of effort from any but the most paranoid corporations (like, as I mentioned, banks). Automation and continuous deployment are definitely the way forward. But even this gargantuan effort is a better option than just "let's deploy and wait for our users to tell us if anything has gone wrong".

But even this gargantuan effort is a better option than just "let's deploy and wait for our users to tell us if anything has gone wrong".

To be fair it sounds like in the original article that they did do some verification that things were working after the deployment. However for some reason their verification tests didn't reveal the presence of a real bug.

Even in a more gargantuan system, it's possible to have tests that give false positive results.

Everyone will screw up releases at some point, the key is to be able to learn from them and get better.

Re: Weekend deployments are for chumps

#22

Post by powdahound »

We use PagerDuty (http://pagerduty.com) at HipChat and while I absolutely loathe being woken up by it, it's helped us identify issues during off-peak hours much more quickly.

But no matter what systems you have in place or how many hundreds of deploys you've done, there's always a new way for things to break.

Re: Weekend deployments are for chumps

#23

Post by krobertson »

Their problem isn't their deployment process, its their monitoring.

Blindly ignoring errors is a recipe for failure. You should always look at situation like that asking "how can we monitor this weak point?" Logging plus a service like Splunk work great.

Should always have a solid on call rotation. We have two rotations, and ops one which is first line, and dev in case deeper code changes or more eyes on it are needed.

Re: Weekend deployments are for chumps

#24

Post by agentultra »

Amen.

I've tried convincing many companies I've worked for that weekend deployments are a bad idea over the years.

Even with continuous integration tests, rolling deployments, and all the precautions in the world things can still happen.

You need live people available to handle a deployment.

Personally, I don't like working on weekends. I've worked for companies that refused to believe that this was a bad idea. I learned pretty fast that life is too short to work on a weekend.

If something does go wrong, it's better to have people on hand to correct the error and get back on track. It's much easier to schedule those people during the work week. It's not rocket science.

Re: Weekend deployments are for chumps

#25

Post by ww520 »

Deploying in weekend or at night is a terrible idea in disguise of a good idea. What we used to do are:

- No deployment on Weekend

- No deployment on Friday

- No deployment after 4pm on Monday to Thursday

- Deployment is rolled out in stage: one server, 5%, 10%, 50%, 100% of servers.

- Rollback steps must be accompanied with deployment steps.

- Verification steps must be specified in the deployment ticket. Verification is done by QA or OPS, other than Dev.

- Common deployment and rollback steps are automated.

- Emergency deployment is an exception to the above but must take extra precaution to babysit the deployment process.

Stress level has gone down a lot and problems are resolved much faster once we have the above.

Re: Weekend deployments are for chumps

#26

Post by tomjen3 »

powdahound wrote: post #22

We use PagerDuty ( http://pagerduty.com ) at HipChat and while I absolutely loathe being woken up by it, it's helped us identify issues during off-peak hours much more quickly. But no matter what systems you have in place or how many hundreds of deploys you've done, there's always a new way for things to break.

I am guessing that HipChat is a startup?

Because as an employee any pager I had would be left at work.

Re: Weekend deployments are for chumps

#27

Post by gecko »

viraptor wrote: post #15

I can't believe in how misguided this post is... 1. They provide a service to people around the world, yet they don't ensure that someone is available as an emergency contact on Sunday evening, when the first post-deployment usage happens. 2. They don't have a universal list of "this breaks, contact that guy". 3. They don't have a known instant rollback procedure for a release. 4. They don't have cross-component inte…

EDIT: I posted a rundown of our deployment process, including where and how tests happen, and why they failed to catch this bug, at http://news.ycombinator.com/item?id=2301680 .

While I'm sure there's a lot of stuff we could improve, the situation's not exactly as you describe.

Responding to a few contacts:

1 & 2. We do have a list of "if this breaks, contact this guy." What we don't have (in response to your first point) is a demand that those people be available Sunday night.

3. We have a known rollback procedure. It does not work if we do a irreversible schema change and the problem's not caught until 20 hours later. We couldn't just throw out 20 hours of data.

4. We actually do a lot of testing. Beginning on Wednesday, we deploy to our early leak accounts. We steadily increase that through the week. The problem with this particular bug is that you could use Kiln lightly (most of our test accounts are not large accounts) without hitting this problem at all. Even the full QA test suite did not trigger the problem. That happened because Kiln was designed to keep working in the case of a FogBugz communication failure until it couldn't, which was directly proportional to how much you used Kiln. The real problem here, which has been fixed, is that Kiln should not attempt to hide a problem communicating with FogBugz.

5. We don't do release in the middle of the day. We do them at 10 PM. I have no idea where you got that.

There's a lot we can improve. We need to make sure Kiln not talking to FogBugz, which can bring down Kiln, hard-fails, instead of trying to continue. We need to make sure that all hands are on-deck when people are going to work, as you noted, which is vastly easier to do midweek than Sunday night. And we probably ought to add more automated testing to the integration points. But I think you're painting a somewhat unfair picture of the current situation.

Re: Weekend deployments are for chumps

#28

Post by gecko »

swombat wrote: post #8

Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...

The releases are both automated (except for one component, as noted, which we are now automating), and are fully vetted.

Here is the old release process:

1. Monday morning, the version to be used for the next release is automatically built for the QA team, who begins running their test suites on it and doing soft checks.

2. By no later than Wednesday, the new version is leaked to testing an alpha accounts on Fog Creek On Demand. Tests are re-run at this point.

3. The leak is increased later in the week if the QA results look good, or the weekend release is canceled, depending on how testing goes.

4. Provided everything has been good, on Saturday night, the leak is increased to 100% of customers. This step does not have a full QA rundown, because the code has already been vetted several times by QA at this point. The sanity checks are truly sanity checks.

5. At the same time, we monitor that our monitoring system (Nagios) agrees that all accounts are online and that there are no major problems, such as massive CPU spikes.

So far, so good. The issue with this release is we had a bug that did not manifest for awhile, because Kiln had been deliberately designed to ignore the failure condition "as long as possible", which ended up just being too damn long. Once we started having failures, we noticed--that's why our sysadmin called us in--but those failures started happening 20 hours after the 100% release, and several days after testing and alpha accounts were upgraded.

I am not arguing our system is perfect, but I'm a nonplussed where the your-deployment-system-totally-sucks stuff is coming from. I'll ask our build manager to post an even more detailed rundown.

Re: Weekend deployments are for chumps

#29

Post by gecko »

sghael wrote: post #19
swombat wrote: post #8

Wait, what? You have large numbers of paying customers to whom you're delivering a mission-critical system (source control isn't exactly optional), and your releases involve neither automated production monitoring/continuous deployment nor formal release procedures? I think your problem is more than just weekend deployments! My full comments here: http://swombat.com/2011/3/8/fog-creek-dont-do-cowboy-deploym...

I agree. My other thought was 'isn't there a staging server in there somewhere?' Something that is near identical to production, with fake production data, etc, that could surface the problem before a customer sees it. btw, props to Fog Creek and OP for airing their dirty laundry. They take some heat, but in the end we all learn from it.

We have more than staging servers: we have staging accounts. I documented our full release process at http://news.ycombinator.com/item?id=2301680.

Re: Weekend deployments are for chumps

#30

Post by dmethvin »

viraptor wrote: post #15

I can't believe in how misguided this post is... 1. They provide a service to people around the world, yet they don't ensure that someone is available as an emergency contact on Sunday evening, when the first post-deployment usage happens. 2. They don't have a universal list of "this breaks, contact that guy". 3. They don't have a known instant rollback procedure for a release. 4. They don't have cross-component inte…

It is so refreshing for any person or company to come out and explain, in detail, how they screwed up and what they are doing to ensure it doesn't happen again.

The tone of your post encourages people to cover up their mistakes for fear of ridicule, and I am against that. Some of your points are worthy of debate though.

Post reply on HN