Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

151–156 of 156 posts

Re: Ask HN: How do you roll back production?

#151

Post by protonimitate »

ksajadi wrote: post #104

Since we run on Kubernetes, rolling back the code is a matter of redeploying the older images. Rolling back database changes is more challenging but usually we have “down” scripts as well as up scrips for all dB changes allowing us to roll database changes back too. We use Cloud 66 Skycap for deployment which gives us a version controlled repository for our Kubernetes configuration files as well as takes care of imag…

Similar set up here... except our CI doesn't handle "down" scripts, so bad db migrations are roll-forward only.

Which.. isn't as bad as I thought it would be (so far).

Re: Ask HN: How do you roll back production?

#152

Post by beebs93 »

sagichmal wrote: post #142
beebs93 wrote: post #113

Earlier quoted context omitted.

Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. The important thing is we increm…

> For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". I've never heard of this practice described as a canary before (shrug)

Yeah, I concede the common lingo within the company (or team even) may not align with either proper definitions or mainstream usage.

Re: Ask HN: How do you roll back production?

#153

Post by beebs93 »

shoo wrote: post #143
beebs93 wrote: post #113

Earlier quoted context omitted.

Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. The important thing is we increm…

> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. this sounds roughly like synthetic monitoring: https://en.wikipedia.org/wiki/Synthetic_monitoring synthetic in the sense of synthetic traffic, s…

> synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Yup - I think that lines up.

> what is the master aggregate "switch" ? what does it do?

We have a hierarchy of aggregrate monitors (or "switches") that watch n amount of either specific metrics or other sub-aggregate monitors.

In the case of production deployments, we watch a specific rollback aggregrate monitor for either a fixed amount of time or customer traffic that will auto-trigger a rollback if it goes into alarm (aka switches on).

We also have a master aggregrate monitor that will switch on if any sub-monitors get swtiched on for any reason. We typically watch this master aggregate alarm to auto-disable any promotions in our code pipeline.

Re: Ask HN: How do you roll back production?

#154

Post by nine_k »

mooreds wrote: post #134
nine_k wrote: post #130

100% of my rollbacks were like this: * Deploy new code in new VMs. * Route some prod traffic to the new nodes. * Watch the nodes misbehave somehow. * Route 100% of the prod traffic back to old nodes (which nobody tore down). Rollback complete. In the case of normal deployment, 100% of prod traffic would eventually be directed to new modes. After a few hours of everything running smoothly, the old nodes would be spun…

So all of your new code was completely backward compatible with your old code (in terms of state)? That is, the database didn't care which version was interacting with it?

Yes, it was backwards-compatible for at least one DB migration step. That is, you can roll forward / back the database schema without breaking other code, or roll forward / back the code without affecting the DB.

This does take more planning and gradual deployment, but saves the day when it matters.

Re: Ask HN: How do you roll back production?

#156

Post by 1ba9115454 »

keithyjohnson wrote: post #41
atemerev wrote: post #19

Blue-green deployment is the only way to fly: https://martinfowler.com/bliki/BlueGreenDeployment.html There are two identical prod servers/cloud configurations/datacenters: blue and green. Each new version is deployed intermittently on blue and green areas: if version N is on blue, version N-1 is on green, and vice versa. If some critical issue happens, rolling back is just switching the front router/balancer to the…

Cost can be a killer here though. If you're flipping from blue to green and vice versa you either have to have capacity in stand-by(expensive) or spin up new capacity before flipping(time-consuming).

We use a blue/green strategy but our infrastructure can be created on demand with Pulumi. https://www.pulumi.com/

It takes Pulumi about 15 minutes to create our kubernetes cluster with all pods and monitoring in place.

After a succesful rollout we can pulumi down one of the clusters and reduce costs (We're on azure.)

Post reply on HN