Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous Deployment at Instagram

engineering.instagram.com

Post reply on HN
Print view View original

51–60 of 94 posts

Re: Continuous Deployment at Instagram

#51

Post by mgorven »

brown9-2 wrote: post #20

What is the purpose of the backlog of deploys? For example, let's say 50 commits all land on master within the same second. Why break those into many deployments stretched across hours instead of deploying them all in the next event? If you landed a bad commit in the middle of that 50, it seems like it might not be immediately obvious once it was deployed that it was bad - and then 5 or 30 minutes later another commi…

By splitting them into multiple deploys it makes it easier to identify bad commits when an error starts. We can usually correlate the start of an error with a specific deploy, and instead of digging through 50 commits to find the cause we only have to look at a few.

Re: Continuous Deployment at Instagram

#52

Post by henrikschroder »

nbm wrote: post #8
iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

Beyond a certain size (basically, once the time the migration will take because of the size of the data it applies to is too large), migrations are a heavy investment - in elapsed time, I/O, and so forth. As such, they are planned to a degree that CD probably isn't the solution for it (for example, you probably can only have one migration in flight at a time). They aren't done live as a single big process that have t…

MySQL has online, non-blocking schema changes since version 5.6. But the underlying data file has to be upgraded to the latest format version for it to work first, and to do that in a non-blocking way on a master server you are probably best off running percona toolkit one first. ALTER TABLE --- FORCE does a data file rewrite.

Re: Continuous Deployment at Instagram

#53

Post by foxylion »

mbesto wrote: post #14
Splendor wrote: post #5

Earlier quoted context omitted.

I've used Flyway for db migrations and it works well. https://flywaydb.org/

Anyone else use Flyway? Looks compelling for anyone not using RoR/Active Record (where migrations are out of box).

We also use it on a multi (postgresql) database setup where every customer has its own database. Works really good, we never had any issues with migrations and do them frequently (maybe once a week).

Re: Continuous Deployment at Instagram

#54

Post by mark242 »

"The test suite needs to be fast. It needs to have decent coverage, but doesn't necessarily have to be perfect."

Holy hell, what a telling statement that is. I get not unit testing for 1 == 1, but come on, unit and integration tests for, say, user login should be difficult, not fast. There are some test suites that actually do need to be perfect, unless Instagram thinks that eg OWASP isn't "decent coverage".

Re: Continuous Deployment at Instagram

#55

Post by briHass »

junto wrote: post #27
daigoba66 wrote: post #3

Earlier quoted context omitted.

A lot of it depends on the particulars of your database system. But there are certainly tools/solutions that exist, and essentially they all boil down to the same pattern: migration scripts should be executed exactly once in order. How this is done varies. I do a lot in the .NET/SQL Server world, and my tool of choice is one that I wrote: http://josephdaigle.me/2016/04/03/introducing-horton.html . Conceptually, what…

Similar tools for the . NET world include Roundhouse and fluentmigrator.net. I'm personally more of a fan of the latter, but we use both in our company in conjunction with Octopus Deploy. Roundhouse needs a bit too much Powershell for my liking.

Octopus Deploy also has a small tool, DBUp, which can run migration scripts. I've been using it for years with no issues.

https://dbup.github.io/

Re: Continuous Deployment at Instagram

#56

Post by txutxu »

yeukhon wrote: post #19

What are the best CD practice for infrastructure? Especially when you have to deal with commits which only need to be in one environment, or commits which need to be in all environments?

Not sure if I understand the question, but for infra, I could say: version it.

Switches have config files or firmware dumps, the same goes for bios and raid bios, for documentation in the infra and connections, etc...

Infra will evolve, and so will do the "version".

While in "test" stage, it's "next version" infra, while in production, the architecture, firmware, connections and configuration, run a tested "version".

Is not easy to integrate/automate infra from different vendors, but it can be done. Been there, done that.

Re: Continuous Deployment at Instagram

#57

Post by kaizendad »

"Expect bad deploys: Bad changes will get out, but that's okay. You just need to detect this quickly, and be able to roll back quickly."

That's an amazing statement to me. I've always worked in smaller environments where we roll up many changes and try to deploy them perfectly. The penalty for bad changes has been high. This is a really new way of thinking.

It's an exciting way of thinking, but I'm not sure I love it. I wonder how well "sometimes we break things" scales with users of smaller services. I guess the flip side is that "we often roll out cool new things" definitely is desirable to users of small services.

Re: Continuous Deployment at Instagram

#58

Post by kaizendad »

madetech wrote: post #24

Shameless Plug: I've recently been involved in writing a book on Continuous Deployment, which covers many of the points Instagram are writing about here (but in greater detail). I've got ~1,000 printed copies to give away. So if anyone wants one, go here: http://madete.ch/1S3OGvl and follow the link on the left hand side and we'll mail a copy to you.

Filled out the form. Definitely excited to get into a CD environment, rather than the 1-2 deploys/day that I've been exposed to in the past!

Re: Continuous Deployment at Instagram

#60

Post by FordPrefectAO »

kaizendad wrote: post #58
madetech wrote: post #24

Shameless Plug: I've recently been involved in writing a book on Continuous Deployment, which covers many of the points Instagram are writing about here (but in greater detail). I've got ~1,000 printed copies to give away. So if anyone wants one, go here: http://madete.ch/1S3OGvl and follow the link on the left hand side and we'll mail a copy to you.

Filled out the form. Definitely excited to get into a CD environment, rather than the 1-2 deploys/day that I've been exposed to in the past!

CD doesn't have to mean 'push every green build to prod'. It's more about the ability to push new functionality when asked by the business, than the fact of always pushing it by default. You may be doing CD well already, knowing just what you have said.
Post reply on HN