Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you roll back production?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 156 posts

Re: Ask HN: How do you roll back production?

#141

Post by seanwilson »

jcampbell1 wrote: post #124
seanwilson wrote: post #88

Earlier quoted context omitted.

Do you have a method to rollback database object removals if they cause problems?

Assuming one step per day to rename a column: create new column, update code to write to new and old, update code to read from new, update code only write to new, rename old to something like xxx_del_pending, delete old column. This process lets you validate and rollback/recover if needed.

Is there a quicker but still fairly safe alternative to this? If you're working solo for example, this doesn't sound practical.

Re: Ask HN: How do you roll back production?

#142

Post by sagichmal »

beebs93 wrote: post #113
sagichmal wrote: post #109

Earlier quoted context omitted.

> Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.) Strange terms. Isn't this just a canary?

Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. The important thing is we increm…

> For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch".

I've never heard of this practice described as a canary before (shrug)

Re: Ask HN: How do you roll back production?

#143

Post by shoo »

beebs93 wrote: post #113
sagichmal wrote: post #109

Earlier quoted context omitted.

> Yeah, very useful strategy. I've heard it go by various names (e.g. toe-dipping, one-boxing, etc.) Strange terms. Isn't this just a canary?

Every company I've been with seems to re-invent the terms or swich their definitions slightly. For me, currently, "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor. The important thing is we increm…

> "canary" means a set of basic automated integration tests that are continually running in production with alarms that feed into a master aggregate "switch". Wether the dedicated canary accounts end up hitting a one-box prod host or real prod host in the end isn't a factor.

this sounds roughly like synthetic monitoring:

https://en.wikipedia.org/wiki/Synthetic_monitoring

synthetic in the sense of synthetic traffic, since it isn't traffic from genuine users.

Can you go into more detail about what is meant by this:

> alarms that feed into a master aggregate "switch"

what is the master aggregate "switch" ? what does it do?

Re: Ask HN: How do you roll back production?

#144

Post by aasasd »

Scarblac wrote: post #137
aasasd wrote: post #91

Earlier quoted context omitted.

Bahahaha, your tests really aren't ‘best’. Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code unless something slips your mind while writing both the code and the tests. Otherwise, it's pretty much ‘deploy and go home.’

It's worse. You need two tests for every time your code should have had an if - maybe the fact that there isn't one is the bug. And if the same unit has multiple branches, you need double the tests to cover all paths. And that doesn't guarantee correctness, and it's only unit tests. But the 50% was a number I vaguely recall from Code Complete.

> every time your code should have had an if

Yeah, I should've probably clarified this, seeing as we're on the topic of correctness and pedantry.

Personally I'm doing this in TDD style, before the code is written. And then while it's written, too. And for code that didn't have tests—the approach really helps catch bugs previously unknown.

Re: Ask HN: How do you roll back production?

#145

Post by karka91 »

Shalle135 wrote: post #96
karka91 wrote: post #89

Assuming this is about web dev. Nowadays - flip a toggle in the admin. Deployments and releases are separated. Made a major blunder? In kubernetes world we do "helm rollback". Takes seconds. This allows for a super fast pipeline and a team of 6 devs pushes out like 50 deployments a day. Pre-kubernetes it would be AWS pipeline that would startup servers with old commits. We'd catch most of the stuff in blue/green phas…

Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.

Honestly, we haven't considered that. Breakage rarely reaches production, rollbacks are quick, a process orchestrator retries most of the stuff and because of that we do not have a problem worth solving with "canary deployments". Note that we do sometimes carry out "canary releases" via toggles.

Re: Ask HN: How do you roll back production?

#146

Post by karka91 »

pojzon wrote: post #110
Shalle135 wrote: post #96

Earlier quoted context omitted.

Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.

How you take care of DB updates when using cannary deployments ? For example those which are not backwards compatible ? Ps. Releases are about building new versions of code packages. Deployments about pushing them out to environments.

This depends a lot on the databases used and flexibility of the code that's accessing the data. One method is to deploy in multiple stages and to use views. In pre-deploy you create a view with schema/data needed for the release. In deploy stage you roll-out the canary code. In post-deploy stage remove either the old data/schema on success or new data/schema on failure. It's quite an overhead to implement and maintain this process.

Re: Ask HN: How do you roll back production?

#147

Post by matwood »

seanwilson wrote: post #141
jcampbell1 wrote: post #124

Earlier quoted context omitted.

Assuming one step per day to rename a column: create new column, update code to write to new and old, update code to read from new, update code only write to new, rename old to something like xxx_del_pending, delete old column. This process lets you validate and rollback/recover if needed.

Is there a quicker but still fairly safe alternative to this? If you're working solo for example, this doesn't sound practical.

I'll answer both your questions here. Yes, we can rollback a db change using migrates but usually a true rollback requires a restore since data will be lost.

In general what I have found is that database changes are unique to your situation. When an app is small it's fine for them to automatically run with the code commits. The system I deal with now has some very large tables, and running database migrations often requires planning. A column addition might be added weeks before the code is written to use the new column. It's just the nature of dealing with large tables.

Working solo, typically also means smaller, so there is a lot more leeway. I would do whatever works for you, and realize it's a good thing if you ever large enough to need to address other problems.

Re: Ask HN: How do you roll back production?

#148

Post by matwood »

mooreds wrote: post #85
matwood wrote: post #66

Earlier quoted context omitted.

Correct. We 'mark' no longer used database objects for removal at some future date. And mark is really just add a ticket to be completed in the future.

How long does it stick around before final removal, typically?

Depends. Table removals are usually quicker because they have zero system impact once all the references in code have been removed. Column removals can require planning depending on the size of the table and if indexes will have to be changed.

Re: Ask HN: How do you roll back production?

#149

Post by lelabo_42 »

We started using docker and kubernetes not long ago. Every deployement in production must have a release number as a tag. If one element of our environment need a rollback, I redeploy an old image on kubernetes. Its very fast, only a few seconds to rollback and you can do rolling updates to avoid downtime.

Re: Ask HN: How do you roll back production?

#150

Post by MauranKilom »

aasasd wrote: post #125
MauranKilom wrote: post #115

Earlier quoted context omitted.

> Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code if (x % 3 == 0) println "Fizz" if (x % 5 == 0) println "Buzz" There, solved it! And it works fine in all cases you told me to test (e.g. 3, 4, and 5), which means it's impossible I botched anything! Surely I nailed this interview…

> all cases you told me to test (e.g. 3, 4, and 5)! Surely I nailed this interview Well, if you don't see other points where the input→output mapping can diverge (and rely on the client's requirements for that?), then no, you didn't nail it.

The point is that you claim to have "the" (best?) way to do proper unit tests, but it still fails in basic cases like some combination of paths through if statements being buggy (even if you properly covered each one in isolation).

You normally cannot prove general correctness with unit tests. You can try to probe interesting points in the input space, and different coverage metrics encourage different levels of rigour and effort with this, leading to different fractions of bugs caught, but you'll never have a guarantee to catch all bugs (short of formal proof or exhaustive input coverage).

Post reply on HN