Continuous Deployment at Instagram • Hacker News Board

Re: Continuous Deployment at Instagram

#71

Post by Rapzid » Wed, Apr 13, 2016, 1:25 AM UTC

henrikschroder wrote: post #52

nbm wrote: post #8

Earlier quoted context omitted.

Beyond a certain size (basically, once the time the migration will take because of the size of the data it applies to is too large), migrations are a heavy investment - in elapsed time, I/O, and so forth. As such, they are planned to a degree that CD probably isn't the solution for it (for example, you probably can only have one migration in flight at a time). They aren't done live as a single big process that have t…

MySQL has online, non-blocking schema changes since version 5.6. But the underlying data file has to be upgraded to the latest format version for it to work first, and to do that in a non-blocking way on a master server you are probably best off running percona toolkit one first. ALTER TABLE --- FORCE does a data file rewrite.

That's a big "yeah but".

You can't rate limit the io. That's huge and the percona tool and LHM offer ways to keep io under control. Along with that you have to pay attention to the live schema update matrix in the docs; many things will upgrade with out locking but require copying the table and this will blast your IO.

In addition even with the tools some changes require exclusive table locks. It needs it for a short period of time but if it can't get it because of a long running transaction , queue of transactions, and etc it can block everything up.

Re: Continuous Deployment at Instagram

#72

Post by kaizendad » Wed, Apr 13, 2016, 1:49 AM UTC

FordPrefectAO wrote: post #60

kaizendad wrote: post #58

Earlier quoted context omitted.

Filled out the form. Definitely excited to get into a CD environment, rather than the 1-2 deploys/day that I've been exposed to in the past!

CD doesn't have to mean 'push every green build to prod'. It's more about the ability to push new functionality when asked by the business, than the fact of always pushing it by default. You may be doing CD well already, knowing just what you have said.

My preconception is that what you said is exactly right, but the idea that one maybe should push every green build to prod -- and the fact that a lot of big-scale companies are doing that -- is really intriguing.

Re: Continuous Deployment at Instagram

#73

Post by ihsw » Wed, Apr 13, 2016, 2:08 AM UTC

serge2k wrote: post #69

nbm wrote: post #31

Earlier quoted context omitted.

I'm not sure about Instagram, but Facebook is a fan of rebasing in general. Nothing should ever appear as a commit in master that isn't something that should be used in production - ie, should never intentionally be broken in isolation. In general, feature branches are relatively very short-lived, and will be code reviewed, rebased and landed as a single commit onto master. Features are often feature flagged off anyw…

> Nothing should ever appear as a commit in master that isn't something that should be used in production - ie, should never intentionally be broken in isolation. I don't understand why people do it any other way.

Some people Ctrl+S and commit every few files, to keep from pushing changes to 10+ files in a single commit.

Re: Continuous Deployment at Instagram

#74

Post by 467568985476 » Wed, Apr 13, 2016, 2:55 AM UTC

mark242 wrote: post #54

"The test suite needs to be fast. It needs to have decent coverage, but doesn't necessarily have to be perfect." Holy hell, what a telling statement that is. I get not unit testing for 1 == 1, but come on, unit and integration tests for, say, user login should be difficult , not fast . There are some test suites that actually do need to be perfect, unless Instagram thinks that eg OWASP isn't "decent coverage".

The probably use an existing, well tested tool for user authentication (why reinvent the wheel?), or have comprehensive coverage for it already. There's no reason for tests for logging in to be slow either way. And after all, it's a social media app, it doesn't have to run perfectly all the time.

Re: Continuous Deployment at Instagram

#75

Post by adamst85 » Wed, Apr 13, 2016, 3:29 AM UTC

madetech wrote: post #24

Shameless Plug: I've recently been involved in writing a book on Continuous Deployment, which covers many of the points Instagram are writing about here (but in greater detail). I've got ~1,000 printed copies to give away. So if anyone wants one, go here: http://madete.ch/1S3OGvl and follow the link on the left hand side and we'll mail a copy to you.

Very interesting! I have signed up for a copy. Thank you

Re: Continuous Deployment at Instagram

#76

Post by mianos » Wed, Apr 13, 2016, 3:35 AM UTC

iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

We use alembic, a python sort of DSL, for migrations. We can go forward or backwards. Any schema changes need an alembic script for both directions. We generally try to do 'mini-migrations' that go with each change, as opposed to big bang migrations. In the past building stock exchanges it was generally big bang with a mass of SQL that could only run once. This are better these days. :)

Re: Continuous Deployment at Instagram

#77

Post by greenleafjacob » Wed, Apr 13, 2016, 3:54 AM UTC

avolcano wrote: post #28

Also seconding the confusion that other commenters have regarding the "three commits max" rule for automated deploys. Maybe engineers at Facebook are just big fans of rebasing, but I often make commits on feature branches that don't "stand on their own" - i.e., would break some functionality without subsequent commits. I'm not sure why you'd want to deploy one-commit-at-a-time unless you kept a very strict "one commi…

The biggest reason IMO is git bisect which is mostly broken without having each commit be "on its own".

Re: Continuous Deployment at Instagram

#78

Post by siculars » Wed, Apr 13, 2016, 4:33 AM UTC

lopatin wrote: post #66

Looks like a lot of schema migration talk here. Out of curiosity does anyone have production experience with lazy migrations for serialized data? Where your model migrations exist as code: an array of functions that convert one version of the model schema to the next. The schema version is encoded into the data. The migrations are lazy because the model is fast forwarded its latest version at the last possible moment…

I work for Basho. Would love to hear more about that use case. Regardless, unstructured data with encoded schema/explicit versioning ftw!

Re: Continuous Deployment at Instagram

#79

Post by HealthyTree » Wed, Apr 13, 2016, 5:00 AM UTC

Regarding canary releases and detecting errors, one aspect that is sometimes overlooked is the possibility for bugs on the client side. At work, we have a fairly large js centric app with a non-negligible amount of bugs pure client side. While tracking http status codes on the backend is fairly straightforward, we find it much harder to get the same type of information from the frontend. Would love to hear if anyone has experience in that area.

Re: Continuous Deployment at Instagram

#80

Post by nevon » Wed, Apr 13, 2016, 5:23 AM UTC

HealthyTree wrote: post #79

Regarding canary releases and detecting errors, one aspect that is sometimes overlooked is the possibility for bugs on the client side. At work, we have a fairly large js centric app with a non-negligible amount of bugs pure client side. While tracking http status codes on the backend is fairly straightforward, we find it much harder to get the same type of information from the frontend. Would love to hear if anyone…

Look into client-side error tracking. We're using Sentry where I work.

It will catch any uncaught exceptions, group them together and normalize the stack traces (because of course they look different across browsers). If you tag the errors by their release, you can see if your release introduced any regressions.

We're not using it in any automated way, though, because there's so much noise. Any time a phone happens to run out of memory or some extension crashes, you'll get an error report.

Of course, it also doesn't cover non-crashing regressions, where you may have incorrect behavior rather than crashes. Those are much harder to catch, unless your integration tests are incredibly granular.