BlueGreenDeployment (2010) • Hacker News Board

Re: BlueGreenDeployment (2010)

#21

Post by neilellis » Fri, Jun 05, 2015, 1:32 PM UTC

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

Personally I keep the database out of the equation, ala 12 Factor.

The database when treated as a unique exception allows you to just get on with developing robust infrastructure.

However for this to work you need to keep code off the database and treat database changes as migrations not blue/green deploys.

I can't say this enough, treat the database as an exception, keep it clean without code on it and with 'soft' schemas (ones that tolerate minor changes well) - then you can do b/g deploys easily.

Also let someone else host the DB where possible (or the same cloud infrastructure of course), it's just not worth the pain.

Re: BlueGreenDeployment (2010)

#22

Post by neilellis » Fri, Jun 05, 2015, 1:34 PM UTC

robinwarren wrote: post #13

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

I believe (in practice if not theory) it is common to handle db schema updates separately and to run both systems off the same database. This way the data only lives in one location but you get to run two slightly different app servers/frontends on top of it. You are then more careful with database changes. This sounds like a pain in the arse but a lot of people seem to be able to run with it without it causing a lot…

Yep agree, also keep the database schema a little soft - i.e. tolerant of minor changes.

( an example for SQL databases of this is having an 'additional info' field/table which has no value for indexing but contains additional captured info.)

Re: BlueGreenDeployment (2010)

#23

Post by neilellis » Fri, Jun 05, 2015, 1:35 PM UTC

dorfsmay wrote: post #18

Some companies will also activate the new path only a small percentage of traffic and monitor the logs to be able to detect errors that were missed in testing.

Yes, I really like that strategy, not done it myself as a little (not a lot!) complicated for a small company.

Re: BlueGreenDeployment (2010)

#24

Post by osipov » Fri, Jun 05, 2015, 1:45 PM UTC

boobsbr wrote: post #20

I work at a big 3-letter multinational corporation, and we still compile and deploy manually. No continuous integration or, at least, automated testing. No build tool whatsoever, or a repo to store artifacts. Everything is compiled on the dev's workstation and sent by email. We do have a disaster recovery server, but it sits outdated for weeks after a deployment in production. When something goes wrong, the switch is…

In a big 3-letter multinational corporation your experience with the build process might not be representative of the corporation at large.

Re: BlueGreenDeployment (2010)

#25

Post by kaylarose » Fri, Jun 05, 2015, 1:54 PM UTC

We use this at the (very large) corp that I work at, with very large, complex, high-traffic/revenue sites. Each component in the architecture has it's own "light/dark" deployment (in each environment), so any piece can be staged/tested/deployed/rolled-back with zero downtime. It has worked fairly well, with the majority of hiccups occuring during the ramp-up on the process.

Re: BlueGreenDeployment (2010)

#26

Post by jbgreer » Fri, Jun 05, 2015, 1:58 PM UTC

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

Yes. Especially if you're using one of the systems as a failover or you're running them as a normally active-active configuration. You want DB replication, but you also want to be able to take down a blue or green, apply DB updates, be able to independently access blue or green to check out your changes, then roll the world over the upgrade leg, then do the same work on the other side, check it out, and then make it available.

Alternating on blue / green as the 'live' side works, too.

Re: BlueGreenDeployment (2010)

#27

Post by pc86 » Fri, Jun 05, 2015, 1:58 PM UTC

dorfsmay wrote: post #17

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

It's complicated and time consuming, but one way to do it is to never modify a column, add new ones and use trigger to copy values from one to the next. Only delete two deploys later.

I'm not saying I have a better solution, but that seems overly complicated and manual (e.g. remembering that in two deploys we're deleting ColumnA and TriggerColAColB) for a system that is seemingly designed to help automate the deployment process.

Re: BlueGreenDeployment (2010)

#28

Post by anton-107 » Fri, Jun 05, 2015, 2:02 PM UTC

robinwarren wrote: post #13

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

I believe (in practice if not theory) it is common to handle db schema updates separately and to run both systems off the same database. This way the data only lives in one location but you get to run two slightly different app servers/frontends on top of it. You are then more careful with database changes. This sounds like a pain in the arse but a lot of people seem to be able to run with it without it causing a lot…

So does this effectively mean - only create tables and add columns in your db schema migrations (no updating and removing)?

Re: BlueGreenDeployment (2010)

#29

Post by danesparza » Fri, Jun 05, 2015, 2:36 PM UTC

That's from 5 years ago.

Also: Your database changes should be entirely backwards compatible and should happen in one location (not on its own slice).

Re: BlueGreenDeployment (2010)

#30

Post by lmm » Fri, Jun 05, 2015, 3:20 PM UTC

anton-107 wrote: post #28

robinwarren wrote: post #13

Earlier quoted context omitted.

I believe (in practice if not theory) it is common to handle db schema updates separately and to run both systems off the same database. This way the data only lives in one location but you get to run two slightly different app servers/frontends on top of it. You are then more careful with database changes. This sounds like a pain in the arse but a lot of people seem to be able to run with it without it causing a lot…

So does this effectively mean - only create tables and add columns in your db schema migrations (no updating and removing)?

You can remove but you update the application to not read from the removed column (and test this in a test environment) first. "Updating" is best done by adding a new column, writing to the new column in parallel, back-populating it, reading from the new column, and then dropping the old column. It's a bit cumbersome but it works and it's safe.