Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous Deployment at Instagram

engineering.instagram.com

Post reply on HN
Print view View original

31–40 of 94 posts

Re: Continuous Deployment at Instagram

#31

Post by nbm »

avolcano wrote: post #28

Also seconding the confusion that other commenters have regarding the "three commits max" rule for automated deploys. Maybe engineers at Facebook are just big fans of rebasing, but I often make commits on feature branches that don't "stand on their own" - i.e., would break some functionality without subsequent commits. I'm not sure why you'd want to deploy one-commit-at-a-time unless you kept a very strict "one commi…

I'm not sure about Instagram, but Facebook is a fan of rebasing in general. Nothing should ever appear as a commit in master that isn't something that should be used in production - ie, should never intentionally be broken in isolation.

In general, feature branches are relatively very short-lived, and will be code reviewed, rebased and landed as a single commit onto master.

Features are often feature flagged off anyway, so it is acceptable to commit partially-functional features to master while that feature is flagged away.

There is a concept of stacked commits, but each commit in the stack needs to be a working step towards the end goal, and as such can (and will) be landed in isolation as they are code reviewed.

Re: Continuous Deployment at Instagram

#32

Post by brianwawok »

krzyk wrote: post #22

Almost all CD cases I've seen talk about canary or black/white(red) releases - a case where there is a fleet of servers. How to do that on a much smaller scale e.g. when I have only 3 servers available? If I deploy to one of them potentially 1/3 of customers might get broken version.

If you paying by the 10 minute increment or less, you just have a 4th server for a few minutes. Something like:

1) Add 4th server at new version

2) Drop 1 old version

3) Launch 1 more new version

4) Drop 1 old version

5) Launch 1 more new version

6) Drop 1 old version

This way you never have less than 3 servers serving requests, but you never pay for more than 4. Should only cost a few cents for most cloud providers for this temp server.

Re: Continuous Deployment at Instagram

#33

Post by brianwawok »

seanwilson wrote: post #16

If they're doing up to 50 commits a day and deploy all commits to master automatically how does that line up with "It makes it much easier to identify bad commits. Instead of having to dig through tens or hundreds of commits to find the cause of a new error, the pool is narrowed down to one, or at most two or three"? If you do a commit and find out in the middle of the day the latest deploy is having problems and peo…

Why roll out to all servers at once? What if each new commit went out to 1% of servers, then spread? At most you need to rollback a small % of servers, as you clearly see which servers are acting bad.

Re: Continuous Deployment at Instagram

#34

Post by perlgeek »

madetech wrote: post #24

Shameless Plug: I've recently been involved in writing a book on Continuous Deployment, which covers many of the points Instagram are writing about here (but in greater detail). I've got ~1,000 printed copies to give away. So if anyone wants one, go here: http://madete.ch/1S3OGvl and follow the link on the left hand side and we'll mail a copy to you.

Hey, I'm currently also writing a book on automating deployments (https://deploybook.com), and if you're interested, I'd like to chat about your experience with writing and publishing this book.

Please drop me a line moritz.lenz@gmail.com. Also, looking forward to read your book :-)

Re: Continuous Deployment at Instagram

#35

Post by daxfohl »

swalsh wrote: post #12
pc86 wrote: post #9

Earlier quoted context omitted.

> The short answer is you design around it. Sorry but that seems like a pretty crap answer. The answer to "what tools allow you to manage database migrations with CD" should not be "don't do database migrations with CD" or "roll your own toggling features."

I apologize for the disappointing answer, but continuously deploying in the sense of constantly pushing code to production multiple times a day isn't just a feature you add to your stack. It's not just another tool. It requires an engineering culture that is focused on it. We have testing, and automated tools for sure, but we have A LOT more tools for logging, and robustly handing errors. When a bad commit is pushed…

If you went back to the previous job, would you push toward more frequent deployments, or is there something that makes the two situations different?

Re: Continuous Deployment at Instagram

#36

Post by UK-AL »

nbm wrote: post #15
hitsurume wrote: post #13

Kind of confused, is this setup for their production deployments only? Does instagram involve any staging environments that would catch a lot of their issues ahead of time, such as failed test builds, bad commits etc? Are their developers allowed to commit directly to master or do they go through a formal pull-request process that gets signed off by someone?

As mentioned in the article itself, Instagram uses code review (using Phabricator), and also uses automated continuous integration for running tests on each "diff", before it allows the change to be landed to master.

Depends what automated tests you run. You could catch things that are technically wrong, but what about things like the button being in the wrong place.

That's nots going to spew errors.

Re: Continuous Deployment at Instagram

#38

Post by crindy »

madetech wrote: post #24

Shameless Plug: I've recently been involved in writing a book on Continuous Deployment, which covers many of the points Instagram are writing about here (but in greater detail). I've got ~1,000 printed copies to give away. So if anyone wants one, go here: http://madete.ch/1S3OGvl and follow the link on the left hand side and we'll mail a copy to you.

Filled out the form. Sending out free physical books seems like a lot though - what's the motivation behind that?

Re: Continuous Deployment at Instagram

#40

Post by nbm »

brazzledazzle wrote: post #37

The post refers to a system called "Landcastle". Is that an internal project or an internal name for an existing open source project?

It's an internal project. It basically takes a "diff" from Phabricator, and goes through the process of applying it to master and ensuring that all unit and integration tests pass before committing it.

This removes the need for every developer to do a rebase, run test, attempt land, discover conflict, rebase, run test, ... loop.

Post reply on HN