Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous Deployment at Instagram

engineering.instagram.com

Post reply on HN
Print view View original

21–30 of 94 posts

Re: Continuous Deployment at Instagram

#21

Post by merb »

nbm wrote: post #8
iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

Beyond a certain size (basically, once the time the migration will take because of the size of the data it applies to is too large), migrations are a heavy investment - in elapsed time, I/O, and so forth. As such, they are planned to a degree that CD probably isn't the solution for it (for example, you probably can only have one migration in flight at a time). They aren't done live as a single big process that have t…

That's not the case for Instagram, they use PostgreSql, they could update tables without any downtime and this is fast. The only problem are migrations which copies data or doesn't just add fields or remove them.

Re: Continuous Deployment at Instagram

#22

Post by krzyk »

Almost all CD cases I've seen talk about canary or black/white(red) releases - a case where there is a fleet of servers.

How to do that on a much smaller scale e.g. when I have only 3 servers available? If I deploy to one of them potentially 1/3 of customers might get broken version.

Re: Continuous Deployment at Instagram

#23

Post by nbm »

krzyk wrote: post #22

Almost all CD cases I've seen talk about canary or black/white(red) releases - a case where there is a fleet of servers. How to do that on a much smaller scale e.g. when I have only 3 servers available? If I deploy to one of them potentially 1/3 of customers might get broken version.

This is actually a problem whether you're using continuous deployment or not. You need to put a release out, which you have done some testing for, but you know that only real production traffic can shake out certain types of problems.

You don't have to have all three servers getting the same amount of traffic, and you don't have to have a single copy of your service on each server. So, you could reduce the weight of a single server that does canary traffic to reduce the pain, or you could run two copies of your service on a server, and have the canary copy get a trickle of traffic.

Another approach is to use shadow traffic - instead of handling the requests on the canary host, you handle it on the production host _and_ the canary host. You'd need to ensure the canary can't adjust the production database, for example - or maybe you only shadow read requests. If you don't get any errors, or you're able to prove to yourself that they function the same, you can then move to a more traditional canary.

You definitely need to adjust your continuous deployment implementation to your environment, whatever it is.

Re: Continuous Deployment at Instagram

#24

Post by madetech »

Shameless Plug: I've recently been involved in writing a book on Continuous Deployment, which covers many of the points Instagram are writing about here (but in greater detail).

I've got ~1,000 printed copies to give away. So if anyone wants one, go here: http://madete.ch/1S3OGvl and follow the link on the left hand side and we'll mail a copy to you.

Re: Continuous Deployment at Instagram

#25

Post by marcc »

krzyk wrote: post #22

Almost all CD cases I've seen talk about canary or black/white(red) releases - a case where there is a fleet of servers. How to do that on a much smaller scale e.g. when I have only 3 servers available? If I deploy to one of them potentially 1/3 of customers might get broken version.

Could you add a 4th (maybe even smaller) server, and configure your load balancer to only route 5% of traffic to this server? We have the same problem, continuously enough requests to implement a system like this, but we only run on a small number of production servers. We've recently added a canary server into the pool, which does increase costs a little, but we made it a small instance, and route only a small amount of traffic to it.

Re: Continuous Deployment at Instagram

#26

Post by ake1 »

mbesto wrote: post #14
Splendor wrote: post #5

Earlier quoted context omitted.

I've used Flyway for db migrations and it works well. https://flywaydb.org/

Anyone else use Flyway? Looks compelling for anyone not using RoR/Active Record (where migrations are out of box).

we use it for a fairly large postgres database (growing 4-5GB a day) and haven't had any big issues so far. currently looking at a sort of catch-22 issue at the moment where the application running flyway also reads information from the database to initialize some caches and that data is inserted and maintained by another application. this works fine on an already initialized database but not for the automated testing server where the data is wiped.

Re: Continuous Deployment at Instagram

#27

Post by junto »

daigoba66 wrote: post #3
iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

A lot of it depends on the particulars of your database system. But there are certainly tools/solutions that exist, and essentially they all boil down to the same pattern: migration scripts should be executed exactly once in order. How this is done varies. I do a lot in the .NET/SQL Server world, and my tool of choice is one that I wrote: http://josephdaigle.me/2016/04/03/introducing-horton.html . Conceptually, what…

Similar tools for the . NET world include Roundhouse and fluentmigrator.net. I'm personally more of a fan of the latter, but we use both in our company in conjunction with Octopus Deploy. Roundhouse needs a bit too much Powershell for my liking.

Re: Continuous Deployment at Instagram

#28

Post by avolcano »

Also seconding the confusion that other commenters have regarding the "three commits max" rule for automated deploys. Maybe engineers at Facebook are just big fans of rebasing, but I often make commits on feature branches that don't "stand on their own" - i.e., would break some functionality without subsequent commits. I'm not sure why you'd want to deploy one-commit-at-a-time unless you kept a very strict "one commit == one standalone feature/bugfix" rule, which isn't mentioned in this post.

(I suppose it's also possible that that's referring specifically to merge commits into master, which would make a lot more sense to me)

Re: Continuous Deployment at Instagram

#29

Post by nbm »

yeukhon wrote: post #19

What are the best CD practice for infrastructure? Especially when you have to deal with commits which only need to be in one environment, or commits which need to be in all environments?

Can you give an example? Your question seems too abstract for me to see if I have any experience with what you're talking about.

Re: Continuous Deployment at Instagram

#30

Post by simonw »

iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

It's hard. The general rule of thumb is that you make database changes independent of code changes, and you always make the changes in such a way that they are compatible with the current deployed code AND with the next version that you plan to roll out.

For example: let's say you have a feature that uses a column, but you want to move it to using a separate table.

Step 1: design the new table

Step 2: deploy the new table - existing code continues to run against the column

Step 3: run a back-fill to ensure the new table has all of the data that exists in the current column

Step 4: deploy code to use the new table instead of the column

The above is the best-case scenario - but it often doesn't work like that, because you need to ensure that data added to the old column between steps 3 and 4 is correctly mirrored across. One approach here is to deploy code that dual-writes - that writes to both the old column and the new table - along with extra processes to sanity-check the conversion.

GitHub's Scientist library is a smart approach to these more complex kinds of migration - which can take months to fully deploy. https://github.com/github/scientist and http://githubengineering.com/scientist/

Post reply on HN