Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous Deployment at Instagram

engineering.instagram.com

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Continuous Deployment at Instagram

#81

Post by kisstheblade »

kaizendad wrote: post #57

"Expect bad deploys: Bad changes will get out, but that's okay. You just need to detect this quickly, and be able to roll back quickly." That's an amazing statement to me. I've always worked in smaller environments where we roll up many changes and try to deploy them perfectly. The penalty for bad changes has been high. This is a really new way of thinking. It's an exciting way of thinking, but I'm not sure I love it…

" I wonder how well "sometimes we break things" scales with users of smaller services" ---

Writing "business software" I have noticed that this doesn't scale at all. I mean when you have a couple of thousand people depending on the software for work bugs are really not tolerated that well.

It's probably different if you have hundreds of servers and can detect bugs on a deployment on one of them, so it only affects a small percentage of users and then you can roll back and try again. But if you have a single installation and you break that all the time with your commits then it probably doesn't work so good. And for the majority of software you really do not need "webscale" installations with millions of "heroku boxen" or droplets etc... Sure have some for redundancy but it really doesn't help with this "deploy master on each commit" type of deal.

Re: Continuous Deployment at Instagram

#82

Post by manyxcxi »

mbesto wrote: post #14
Splendor wrote: post #5

Earlier quoted context omitted.

I've used Flyway for db migrations and it works well. https://flywaydb.org/

Anyone else use Flyway? Looks compelling for anyone not using RoR/Active Record (where migrations are out of box).

We use Flyway in a number of ways. Scripted as part of some deployment processes, included with Spring Boot applications for a RoR style migration on boot, and actually use it in Java code to manage other databases from our application during runtime.

An interesting way to use it is that you can have multiple applications that may use the same schema but are responsible for their own tables- you can specify a different schema metadata table for each application and they can generally live together fairly happily. We also have an application that 'deploys' data to target databases, so we are able to run Flyway via their Java library to make sure the target schemas are correct before running our statements against them.

Prior to Flyway I used Liquibase, which is also pretty powerful- but Flyway has just been so much more versatile- and the Spring Boot 'auto' integration has been awesome.

Re: Continuous Deployment at Instagram

#83

Post by MrGando »

madetech wrote: post #45
lpbonenfant wrote: post #41

Earlier quoted context omitted.

Do you ship outside the United States?

We're based in the UK and have shipped plenty overseas so far. I'm sure we'll be able to get something to you, if you're not in too remote a location.

Can't seem to find the link on mobile safari, care to share in a reply? (I'm in SF)

Re: Continuous Deployment at Instagram

#84

Post by wolframarnold »

kaizendad wrote: post #57

"Expect bad deploys: Bad changes will get out, but that's okay. You just need to detect this quickly, and be able to roll back quickly." That's an amazing statement to me. I've always worked in smaller environments where we roll up many changes and try to deploy them perfectly. The penalty for bad changes has been high. This is a really new way of thinking. It's an exciting way of thinking, but I'm not sure I love it…

That depends on the service. If you can afford outages that may be fair game. But if you have a high traffic service that's running hundreds or thousands of hosts, you can't take them all offline at once. Deploys can take hours, so can rollbacks. In that situation with high SLA requirements you can't really "expect" bad deploys.

Re: Continuous Deployment at Instagram

#85

Post by salex89 »

madetech wrote: post #45
lpbonenfant wrote: post #41

Earlier quoted context omitted.

Do you ship outside the United States?

We're based in the UK and have shipped plenty overseas so far. I'm sure we'll be able to get something to you, if you're not in too remote a location.

Strange, I'm not from the US, (I'm from Serbia) and it says I can't even buy the book on Amazon (I was hoping for the kindle edition). I did apply for the hard copy though.

Re: Continuous Deployment at Instagram

#86

Post by ed_blackburn »

I've always found that the biggest hurdle with CD is never the tech. It's overcoming fear and tweaking culture.

Nevertheless it's always great to read how others accomplish their goals and even better that they're willing to share the journey.

Personally I find it incredibly frustrating to see code that I write not ship for sometimes weeks or even months at some clients. It's a slow process but we'll get there..

Re: Continuous Deployment at Instagram

#87

Post by ed_blackburn »

simonw wrote: post #30
iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

It's hard. The general rule of thumb is that you make database changes independent of code changes, and you always make the changes in such a way that they are compatible with the current deployed code AND with the next version that you plan to roll out. For example: let's say you have a feature that uses a column, but you want to move it to using a separate table. Step 1: design the new table Step 2: deploy the new…

Obviously your millage will vary but I've also found something that augments the above approach is to separate reads from writes in your app. Read from views instead of directly from the tables. You may also choose to write using sprocs, and, or explicit 'writer' types/functions in your app code.

This can help in the migration process because you can version the views and sprocs whilst running more than one version at a time. Essentially you're creating a versioned API for you db.

Of course you now have more db objects to manage (boo, hiss, more moving parts) but it also encourages you down a saner path of versioning your db objects and rationalising your persistence somewhat (Do we really need 3NF? If I update this table in isolation..I jeopardise consistency of this entity etc.)

None of this _solves_ anything but I've found it mitigates a hard problem.

Re: Continuous Deployment at Instagram

#88

Post by ed_blackburn »

swalsh wrote: post #4
iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

I work in a company that does several deployments a day, and has a giant database. The short answer is you design around it. If you can do something without majorly changing your data you do it. Another common thing is to deploy the code, but then to add a "feature toggle" so you just turn the code on when the data is ready. Basically, figure out how you can refuel while in the air.

I did a gig whereby the databases were HUGE changing the schema was aways avoided unless it was deemed it would pay for itself. i.e. the accompanying feature warranted a change and to what degree. To reduce entropy a full time DBA would tweak indexes and work with each component team to release db refactoring with the teams.

I don't think this is an approach I would recommend but if your gig is a giant enterprisey shop that manages risk far to conservatively it's a reasonable half-measure.

CD is an aspiration for a lot of shops and getting there is a cautious tale of lots of small victories and earning the trust of the decision makers. Sometimes that means:

> designing around it

Re: Continuous Deployment at Instagram

#89

Post by ed_blackburn »

iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

Irrespective of language / platform does anyone ever write 'down' migrations? Do you ever use them?

I've always found they don't pay for themselves in terms of testing and likelihood of being required.

Alternatives are snapshots (depending on if you can afford downtime) and simply writing recovery scripts that aren't tied to individual migrations but rather a deployment. i.e. run script before running any migrations and if it goes wrong run contra script to backout as opposed to cleanup script.

Re: Continuous Deployment at Instagram

#90

Post by decode »

iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

A somewhat similar question: what do you do with frontend assets? If you have JS/CSS/image files with a cache-busting URL, and you have different versions on different servers, you can run into problems.

For example, if most of your web servers are running version 1 and you're in the middle of rolling out version 2. A page request goes to one of the web servers with version 2, which returns HTML containing links to assets with the version 2 cache-busting URLs. The browser requests those assets, perhaps through the CDN, but the load balancer sends the requests to a web server still running version 1, where the assets don't exist yet. This means the browser will get a 404 error.

What are the best options for dealing with this problem?

Post reply on HN