Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BlueGreenDeployment (2010)

martinfowler.com

Post reply on HN
Print view View original

11–20 of 38 posts

Re: BlueGreenDeployment (2010)

#11

Post by joshribakoff »

The way I do it is to rsync my code to a path such as "app-v1.0.0"

I then create symlinks such as "live" & "beta":

"live" -> "app-v1.0.0"

I rsync the code from a CI server or local host, after running bower/composer & testing, this way I deploy a "snapshot" of the entire codebase.

I find a lot of people just casually deploy their code with git, and subsequently run "bower install", tolerating downtime in between, with no solid rollback mechanism in place. I hear Capistrano works in a similar way but is git based instead of rsync. To me its really important that the updates happen atomically & roll back atomically, and as a result of running a single command.

Simple example of a "deploy.sh" I would use in a new project:

gulp production && rsync --update-after --delay-updates --exclude-from="rsync_exclude.txt" -avr ./ server-www:/var/www/html/

Re: BlueGreenDeployment (2010)

#12

Post by joshribakoff »

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

Copy DB from live [blue] to staging [green], then run migrations and test. Do this several times. This should give you confidence that the migrations behave as intended before you flip from blue to green for production & run the migrations for real. If you have "big data" you can use schemaless migrations, for example never rename a column, instead add a new column, update the app to deal with both columns, then in a subsequent update remove the old column. You can also avoid migrations [use mongo instead of relational DB, for example]. I recommend Fowler's "DB Refactoring" book for further reading. http://martinfowler.com/books/refactoringDatabases.html

Re: BlueGreenDeployment (2010)

#13

Post by robinwarren »

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

I believe (in practice if not theory) it is common to handle db schema updates separately and to run both systems off the same database. This way the data only lives in one location but you get to run two slightly different app servers/frontends on top of it. You are then more careful with database changes. This sounds like a pain in the arse but a lot of people seem to be able to run with it without it causing a lot of additional overhead having to schedule your db changes to go out before your dependant code changes.

Re: BlueGreenDeployment (2010)

#15

Post by hartror »

robinwarren wrote: post #13
patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

I believe (in practice if not theory) it is common to handle db schema updates separately and to run both systems off the same database. This way the data only lives in one location but you get to run two slightly different app servers/frontends on top of it. You are then more careful with database changes. This sounds like a pain in the arse but a lot of people seem to be able to run with it without it causing a lot…

Yeah this is how you do it. You can also use feature toggles to help manage this. http://martinfowler.com/bliki/FeatureToggle.html

Re: BlueGreenDeployment (2010)

#16

Post by dorfsmay »

madeofpalk wrote: post #6

BlueGreen deployments are easy to achieve using something like AWS ElasticBeanstalk (and they also force you to have your entire env setup in code so its quickly duplicatable for each deploy) with how easy it is to create a new stack and its 'Swap CNAME' feature. However, now that Elastic Beanstalk supports incremental rollouts I don't see as much value in the approach. We use Blue/Green deployments (with AWS EB) at…

Same here, we went to blue/green deploys when magic in Beanstalk broke and created an outage (we used to use beanstalk deploys where it does all the switching for you).

It is very sad and unfortunate that Amazon decided to limit the smaller number of instances to 1 in VPCs. In the old classic AWS you could set the minimum number of instances to 0, which allowed you to completely shutdown an environment without destroying it, effectively allowing you to turn back to it in a few seconds.

Re: BlueGreenDeployment (2010)

#17

Post by dorfsmay »

patsplat wrote: post #10

Database is the sticky part of this strategy. If Blue is in production and Green is on deck, how are updates to the Blue database pushed to Green?

It's complicated and time consuming, but one way to do it is to never modify a column, add new ones and use trigger to copy values from one to the next. Only delete two deploys later.

Re: BlueGreenDeployment (2010)

#19

Post by allan_s »

(looking for comments)

after reading that articles (some months ago) I've decided to implement that in our company , we're mostly using symfony2 and postgresql, and I came up with that https://github.com/allan-simon/ansible-docker-symfony2-vagra...

basically I have

          ______nginx__phpfpm+code  (blue)
         /                        \
   front nginx                      database 
         \______nginx__phpfpm+code/  (green)
The front nginx play the role of "switch" between blue and green , and the database is shared

so basically for the database problem I solve by telling our dev to be aware of it, which translate into a two step process for database upgrade. (we use Dotrine's ORM with doctrine migrations for migrations, so when i talk about getter /setters I talk about the entities' )

  * if we add a column which must be "not null" , we add it with a default value first , so the few second the old code is still online , but the database is already updated , if an insert happen it will not fail . Then second time we drop that default value and put in the migration the SQL statement that handle previous data.

  * if we drop a column,  we first commit a version of our application which remove the code using this column , or make them return default value. and then the second deployement drop the column, so that the version N-1 is already made to not care about it.
etc. the base logic is "the database version N+1 should still work with version N of the code" , I like to image that with a "crossing a small river by always keeping a feet on earth rather than jumping the two feet at once "

after I have to admit that fortunately 99% of our deployment are not about database changes so this "need to do it carefully" only happen once in a while and it does not affect our productivity (especially with the gain of being able to do CI)

last thing, is that I've used this technique only on small to medium websites, nothing with HUGE traffic, so I don't know if it's 100% no downtime, my test (running several `while true; curl WEBSITE ; done ` ) shown no traffic lost.

I just wanted to say that to get opinion from more experienced people, while telling to "normal company" fellows that it's possible to achieve that without "the cloud" and old-school server hosted in a DC or at customer's office.

Edit: of course we still have a dev and staging environment on which we validate code before deploying in production.

Re: BlueGreenDeployment (2010)

#20

Post by boobsbr »

I work at a big 3-letter multinational corporation, and we still compile and deploy manually. No continuous integration or, at least, automated testing. No build tool whatsoever, or a repo to store artifacts. Everything is compiled on the dev's workstation and sent by email.

We do have a disaster recovery server, but it sits outdated for weeks after a deployment in production. When something goes wrong, the switch is not automatic and transparent to the user, we have to email EVERYONE in the user's list to please use the DR server's address. There is no router or load balancer.

Crazy.

Post reply on HN