Continuous Deployment at Instagram • Hacker News Board

Re: Continuous Deployment at Instagram

#91

Post by traviscj » Wed, Apr 13, 2016, 3:20 PM UTC

decode wrote: post #90

iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

A somewhat similar question: what do you do with frontend assets? If you have JS/CSS/image files with a cache-busting URL, and you have different versions on different servers, you can run into problems. For example, if most of your web servers are running version 1 and you're in the middle of rolling out version 2. A page request goes to one of the web servers with version 2, which returns HTML containing links to a…

Not sure that this is the best answer, but seems like a good use of feature flags. If the flag points to the old state, serve the old assets. If it points to the new state, serve the new assets. So make a deploy with both assets, deploy everywhere, then flip the flag.

A bit unsatisfying because it probably isn't always easy the include both assets.

Another idea (not sure if a good one) would be having the load balancers pin a certain session to a certain backend machine. Seems like this would make it better without fixing it, though: that session will still need to switch to a different set of assets when "their" server is deployed.

Re: Continuous Deployment at Instagram

#92

Post by nbm » Wed, Apr 13, 2016, 5:28 PM UTC

decode wrote: post #90

iyn wrote: post #2

What are the best practices for database migrations when trying to setup continuous deployment? Are there any existing tools/solutions that solve/simplify the problem? This is the issue that is almost always missing in articles/tutorial about CD

A somewhat similar question: what do you do with frontend assets? If you have JS/CSS/image files with a cache-busting URL, and you have different versions on different servers, you can run into problems. For example, if most of your web servers are running version 1 and you're in the middle of rolling out version 2. A page request goes to one of the web servers with version 2, which returns HTML containing links to a…

The underlying problem is the use of a naive cache buster which would cause the old server to reply one way and the new one another. For both the people who want the old asset and the new asset.

If instead of being something the web server throws away before replying, the version number actually caused different assets to be returned, then you'd not have this problem.

One pattern is to separate out assets into a different package that is deployed to a separate host group and have your clients request a different host name, or have your load balancers use a path match to use those servers for those requests.

Another is to push asset updates first to all hosts. All hosts, even without code update, will now be able to respond for the new assets.

Another is to use a local cache plus some backend service or database to serve the assets from the web servers - again, all hosts will now respond correctly for the old assets and the new.

Re: Continuous Deployment at Instagram

#93

Post by _opiu » Wed, Apr 13, 2016, 9:33 PM UTC

poli wrote: post #49

madetech wrote: post #48

Earlier quoted context omitted.

It was originally a content marketing experiment. We thought we'd rather have 100 people receive a free printed copy of the book and love us for this, than a 1000 people receive ebooks and end up in their trash. The ROI from the 100 trial worked well, so we decided to try and scale it out to more people. TBD whether we'll see a good ROI from 1000 people, but we hope people enjoy it and like us for sending them a free…

I ordered a copy from Buenos Aires, Argentina. Will you ship it? :)

I was going to order from Nicaragua, but decided to use a U.S. address after all

Re: Continuous Deployment at Instagram

#94

Post by daxfohl » Fri, Apr 15, 2016, 1:51 PM UTC

swalsh wrote: post #47

daxfohl wrote: post #35

Earlier quoted context omitted.

If you went back to the previous job, would you push toward more frequent deployments, or is there something that makes the two situations different?

If I was starting a new company I would without a doubt push towards doing frequent deployments all day long. That said, I would would never push for it at my previous company. At this company it's something they've been doing since day 1. It's always been built into our process, culture, and hiring practices. At my previous company the "agile" process was just a shortened waterfall process. Though many of the engine…

I definitely like that the more I think about it. We try to do deployments every couple months, but changes are so big and risky we end up pushing off deployments to the next release just because nobody wants to risk it.