Testing with Jenkins, Ansible and Docker

Over what? Are you seeking a comparison with releasing one a day, a week, a month, a year?

One of the responses to your question contrasts multiple-times-daily releases with once-every-two-months releases. There are certainly clear benefits there. Personally, i am less convinced of the net benefits of multiple-times-daily releases over a once-a-day or even once-a-week releases.

I work on a few apps which have fast release cycles, where we often release several times a week, and occasionally release several times a day.

One big benefit is the ability to fix bugs fast. When a customer service rep comes over and tells us about a mistake in the data we are showing our users, or a sysadmin comes up and tells us about some alerts we're causing on his dashboard, we can quite often fix the bug and release the fix the same day. That really helps keep customers and sysadmins happy.

Another benefit is the ability to get metrics into production fast. If we're making plans, sometimes we realise that we really need more information about the behaviour of the system or users to make a good decision. If that's information that can be captured in metrics or log events, we can add the metric release, and then gather information quickly enough not to completely derail our planning process.

The ability to add metrics fast also helps with tracking down bugs, because we can essentially instrument the production system on demand. For example, at one point, we had a suspicion that some service that took two overlapping date ranges as parameters was being called with date ranges that did not overlap. So we added a check for that which logged an event if they didn't, and pushed it to production. The next morning, we knew exactly how much this was happening, and which other application the was making the calls.

The flip side of this is that our frequently-released apps get essentially no meaningful manual QA, and precious little automatic testing. We have huge suites of tests which we run in CI, of course, but we don't have time to put every release through soak or performance tests.

If you wanted to make a general handwavy statement, i'd say that having really fast release cycles helps break down the wall between development and production. Which you might or might not find was a good thing.

But of course there's a limit; you don't want releases that are so quick that you don't collect enough data about the effects of each release.

It's specifically the motivation for continuous deployment that I'm curious about here. Fixing bugs quickly is obviously a plus, and I found the point 'twic mentioned about rapidly instrumenting a production system interesting. These are good arguments for being able to release an update on demand, and indeed for continuous integration as a technique to support that goal.

However, to me neither seems like a strong argument for routinely releasing multiple updates per day. I've always been a little surprised that so many web sites and SaaS applications seem to take pride in making such frequent changes, even though doing so surely incurs some risk of breaking things and typically also causes some degree of instability in the UI. I was just wondering whether anyone had experiences and/or actual data they could share about whether the practice made a measurable improvement to, for example, revenues or reported customer satisfaction -- something concrete that might outweigh the risks -- or whether it's more of a subjective preference for that way of working.

I don't understand this line of thinking. If there's a bug that you can fix in short order, why not fix it and deploy said fix? If you've set your CI and infrastructure up correctly, update rollouts aren't something the user even notices.

I've always been a big proponent of small, atomic commits. Being able to deploy just one thing at a time means we often know where breakage comes from (when it happens), and we can respond to feedback sometimes within minutes.

The customers love it, and we enjoy excellent stability and development velocity. These are the reasons we deploy many times a day. There's no reason for us not to deploy things once they are tested and ready.

This was a major, major issue with puppet, too - although somewhat worse, in that the manifests were more unreadable.

One issue that ansible still hasn't really solved is the horrible if/then/elses you have to put in to accommodate RHEL/ubuntu/centos/whatever else differences.

I like that galaxy at least lets you specify which environment the playbook will run on, though.