Deploys at Slack • Hacker News Board

Re: Deploys at Slack

#41

Post by tantalor » Wed, Apr 08, 2020, 10:02 PM UTC

nathankunicki wrote: post #37

Fun to read, but there's a lack of detail here that I'd like to see. For example, this talks purely about code changes. However times a code change requires a database schema change (as mentioned above), different API's to be used, etc. In the percentage based rollout where multiple versions are in use at once, how are these differences handled?

Easy: don't do that.

Always make your code compatible with the old and new schema. Migrate the database separately. Then after the migration, remove the code that supports the old schema.

Re: Deploys at Slack

#42

Post by tantalor » Wed, Apr 08, 2020, 10:04 PM UTC

shusson wrote: post #6

What happens after hot-fixing the release branch? Does the release branch get merged back into master?

Usually you fix master first then cherrypick the fix to the release branch.

Re: Deploys at Slack

#43

Post by navaati » Wed, Apr 08, 2020, 10:08 PM UTC

nathankunicki wrote: post #37

Fun to read, but there's a lack of detail here that I'd like to see. For example, this talks purely about code changes. However times a code change requires a database schema change (as mentioned above), different API's to be used, etc. In the percentage based rollout where multiple versions are in use at once, how are these differences handled?

For database schema changes, here is the standard practice: - You have version 1 of the software, supporting schema A. - You deploy a version 2 supporting both schema A and new schema B. Both versions coexist until the deployment iis complete and all version 1 instances are stopped. During all this time the database is still on schema A, this is fine because your instances, both version 1 and 2, support schema A. - Now you do the schema upgrade. This is fine because your instances, now all runnning version 2, support schema B - At last, if you wish you can now deploy a version 3, dropping the support for schema A.

Re: Deploys at Slack

#44

Post by aledalgrande » Wed, Apr 08, 2020, 10:22 PM UTC

A few questions I have left unanswered:

- does the deploy commander create the hotfixes or the engineers who authored the commits?

- it seems that the deployment is fully automated, but engineers still have to be available in case of problems, does that impact productivity?

- "Once we are confident that core functionality is unchanged", is there a particular metric to assert that?

- how long does deployment take currently?

- switching directories doesn't seem like a fully atomic operation yet, isn't there a delay from loading the files and wouldn't that generate 502s from the service? Maybe it's better to create new instances with the new files and then change the router to use those (blue-green)?

Re: Deploys at Slack

#45

Post by dang » Wed, Apr 08, 2020, 10:43 PM UTC

7ewis wrote: post #23

This link has now been reposted 6 times in the past two weeks: https://news.ycombinator.com/item?id=22816645 https://news.ycombinator.com/item?id=22729766 https://news.ycombinator.com/item?id=22801191 https://news.ycombinator.com/item?id=22784712 https://news.ycombinator.com/item?id=22720028 https://news.ycombinator.com/item?id=22806810

That's an indicator of interest. I actually emailed one of the submitters to repost the article for that reason. (Yes, we're thinking about software to detect cases like this.)

On HN, a submission doesn't count as a dupe unless it has had significant attention. This is in the FAQ: https://news.ycombinator.com/newsfaq.html.

Re: Deploys at Slack

#46

Post by ivanfon » Wed, Apr 08, 2020, 10:48 PM UTC

mrdonbrown wrote: post #27

brycethornton wrote: post #2

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

Sleuth is a SaaS deployment tracker that pulls deployments from source repositories, feature flags, and other sources, in addition to pushes via curl. You can see Sleuth used to, well, track Sleuth at https://app.sleuth.io/sleuth [Disclaimer: am a Sleuth co-founder]

Is it possible to view the page you linked without creating an account? It redirects me to your landing page.

Re: Deploys at Slack

#47

Post by Jestar342 » Wed, Apr 08, 2020, 11:02 PM UTC

brycethornton wrote: post #2

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

I don't know if this will tick all of the boxes you need because it is primarily IAC, and is for k8s only afaik: https://www.pulumi.com

Re: Deploys at Slack

#48

Post by aledalgrande » Wed, Apr 08, 2020, 11:04 PM UTC

yoloClin wrote: post #40

nathankunicki wrote: post #37

Fun to read, but there's a lack of detail here that I'd like to see. For example, this talks purely about code changes. However times a code change requires a database schema change (as mentioned above), different API's to be used, etc. In the percentage based rollout where multiple versions are in use at once, how are these differences handled?

I'm more curious about how DB rollbacks occur in situations where a PR changes DB and is then reverted.

It would be a good practice to first make a DB change alone, which is compatible with both and new code, so you don't need rollbacks. Then separately deploy a code change.

Edit: also suggested by Martin Fowler https://www.martinfowler.com/bliki/BlueGreenDeployment.html

Re: Deploys at Slack

#49

Post by aledalgrande » Wed, Apr 08, 2020, 11:05 PM UTC

tantalor wrote: post #41

nathankunicki wrote: post #37

Fun to read, but there's a lack of detail here that I'd like to see. For example, this talks purely about code changes. However times a code change requires a database schema change (as mentioned above), different API's to be used, etc. In the percentage based rollout where multiple versions are in use at once, how are these differences handled?

Easy: don't do that. Always make your code compatible with the old and new schema. Migrate the database separately. Then after the migration, remove the code that supports the old schema.

I think every DB change should be done like you suggest. An example I worked on recently:

- migrate DB and create new field

- deploy code for writing into such field (not read yet), in parallel with old field

- backfill data migration for older records

- deploy code with feature flag to read new field in workflows, but still write to both fields

- switch read feature flag on

- make sure everything works for a few weeks

- switch write feature flag to only use new field

Re: Deploys at Slack

#50

Post by joshgachnang » Wed, Apr 08, 2020, 11:10 PM UTC

How are deploy commanders chosen? On my (very small currently) team, the person who is on-call is also our deploy commander, but it seems like you might need something else for a larger team.