I work in a company that does several deployments a day, and has a giant database. The short answer is you design around it. If you can do something without majorly changing your data you do it. Another common thing is to deploy the code, but then to add a "feature toggle" so you just turn the code on when the data is ready. Basically, figure out how you can refuel while in the air.
> The short answer is you design around it. Sorry but that seems like a pretty crap answer. The answer to "what tools allow you to manage database migrations with CD" should not be "don't do database migrations with CD" or "roll your own toggling features."
How is lack of extant tooling swalsh's fault? Would you like to write some?
I work in a company that does several deployments a day, and has a giant database. The short answer is you design around it. If you can do something without majorly changing your data you do it. Another common thing is to deploy the code, but then to add a "feature toggle" so you just turn the code on when the data is ready. Basically, figure out how you can refuel while in the air.
> The short answer is you design around it. Sorry but that seems like a pretty crap answer. The answer to "what tools allow you to manage database migrations with CD" should not be "don't do database migrations with CD" or "roll your own toggling features."
I apologize for the disappointing answer, but continuously deploying in the sense of constantly pushing code to production multiple times a day isn't just a feature you add to your stack. It's not just another tool. It requires an engineering culture that is focused on it. We have testing, and automated tools for sure, but we have A LOT more tools for logging, and robustly handing errors. When a bad commit is pushed into production, I can pull up a graph that shows me the number of errors, and the full read out of every occurrence of it. I can also revert that code very quickly. We also have a team that does nothing but monitor the code, and the deployments. There are a bunch of tools you can find probably... but my point is that it's something we're thinking about from the design stage... because from design to production it might just be a few days. Major features will get pushed in stages, its a very different way of doing things (at least for me). Before I was here, deploying was something I did maybe once every few months. The code had to be absolutely perfect, because it'll be a while before I can get a chance to fix it. Here, I deploy while I drink my morning coffee.
Kind of confused, is this setup for their production deployments only? Does instagram involve any staging environments that would catch a lot of their issues ahead of time, such as failed test builds, bad commits etc? Are their developers allowed to commit directly to master or do they go through a formal pull-request process that gets signed off by someone?
What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD
I've used Flyway for db migrations and it works well. https://flywaydb.org/
Anyone else use Flyway? Looks compelling for anyone not using RoR/Active Record (where migrations are out of box).
Kind of confused, is this setup for their production deployments only? Does instagram involve any staging environments that would catch a lot of their issues ahead of time, such as failed test builds, bad commits etc? Are their developers allowed to commit directly to master or do they go through a formal pull-request process that gets signed off by someone?
As mentioned in the article itself, Instagram uses code review (using Phabricator), and also uses automated continuous integration for running tests on each "diff", before it allows the change to be landed to master.
If they're doing up to 50 commits a day and deploy all commits to master automatically how does that line up with "It makes it much easier to identify bad commits. Instead of having to dig through tens or hundreds of commits to find the cause of a new error, the pool is narrowed down to one, or at most two or three"?
If you do a commit and find out in the middle of the day the latest deploy is having problems and people are still committing in new code wouldn't this make things much harder to narrow down?
If they're doing up to 50 commits a day and deploy all commits to master automatically how does that line up with "It makes it much easier to identify bad commits. Instead of having to dig through tens or hundreds of commits to find the cause of a new error, the pool is narrowed down to one, or at most two or three"? If you do a commit and find out in the middle of the day the latest deploy is having problems and peo…
Assuming they're spread out over ~10-12 hours (some crazy morning people, some crazy nocturnal people), that's only ~4-6 commits per hour.
Most problems will be discovered by someone and reported in an hour, and most of those will also be discoverable in a dataset on a system like Scuba - https://www.facebook.com/notes/facebook-engineering/under-th... - and you can identify the first time that particular issue happened.
If you're lucky, it lines up exactly to a commit landing, and you only need to look at that. Otherwise, due to sampling, maybe you need to look at two or three commits before your first report/dataset hit. You can also use some intuition to look at which of the last n commits are the likely cause. A URL generation issue? Probably the commit in the URL generation code. You'd do the same thing with a larger bundled rollout, but over a larger number of commits (50, in the case of a daily push).
What are the best CD practice for infrastructure? Especially when you have to deal with commits which only need to be in one environment, or commits which need to be in all environments?
For example, let's say 50 commits all land on master within the same second. Why break those into many deployments stretched across hours instead of deploying them all in the next event?
If you landed a bad commit in the middle of that 50, it seems like it might not be immediately obvious once it was deployed that it was bad - and then 5 or 30 minutes later another commit is deployed on top of it.
You might not notice a problem until hours after all of the commits have been deployed, which leaves you in the same situation as if you had deployed all 50 changes in one event, but in this model those 50 commits have been stretched over a much longer period of time between commit and liveness to users.