Live data from Hacker News

Deploys at Slack

slack.engineering

111–120 of 139 posts

Re: Deploys at Slack

#111
post #107

Earlier quoted context omitted.

If I was using PHP, I wouldn't use containers either. Just sync the latest code over, change a sym link to the new build, done.

capistrano is perfect for that. we use it for all our deployment needs and it has been wonderful!

second this. I haven't needed cap in a while for the work I'm doing, and I don't often see it mentioned (perhaps because I'm not looking) but it's a fantastic tool for managing atomic deployment.

Re: Deploys at Slack

#112
Their deployment UI looks nice but this feels like they made their own wheel here in order to keep their In-place upgrade method over something such immutable infrastructure using pre-existing deployment systems.

I wonder if this was ruled out for some reason or perhaps for a large company with people dedicated to deploying this doesn't matter. One example, as they are on AWS autoscaling groups with prebuilt AMI's could have been used to roll new machines instead of copying files to the server.

Re: Deploys at Slack

#113
post #43

Fun to read, but there's a lack of detail here that I'd like to see. For example, this talks purely about code changes. However times a code change requires a database schema change (as mentioned above), different API's to be used, etc. In the percentage based rollout where multiple versions are in use at once, how are these differences handled?

For database schema changes, here is the standard practice: - You have version 1 of the software, supporting schema A. - You deploy a version 2 supporting both schema A and new schema B. Both versions coexist until the deployment iis complete and all version 1 instances are stopped. During all this time the database is still on schema A, this is fine because your instances, both version 1 and 2, support schema A. - N…

We do it the other way (and I’ve always seen it done this way): database change is compatible with current code and new code. So deploy the database change, then deploy the code change. It usually allows you to rollback code changes.

Re: Deploys at Slack

#114

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.

Please provide a way for people to reach you without commenting here.

Re: Deploys at Slack

#115
post #43

Earlier quoted context omitted.

For database schema changes, here is the standard practice: - You have version 1 of the software, supporting schema A. - You deploy a version 2 supporting both schema A and new schema B. Both versions coexist until the deployment iis complete and all version 1 instances are stopped. During all this time the database is still on schema A, this is fine because your instances, both version 1 and 2, support schema A. - N…

We do it the other way (and I’ve always seen it done this way): database change is compatible with current code and new code. So deploy the database change, then deploy the code change. It usually allows you to rollback code changes.

This is generally harder to pull off though unless you do things like force all DB access to go through stored procedures.

And then you're really still pursuing the same strategy described above, except for your stored procedures instead of your app code.

Re: Deploys at Slack

#116

It's cool to see how big organizations have deployment setups, while it feels like there is not enough resources about how one should setup a deployment system for a new startup just in the beginning. The setup I currently use is custom bash scripts setting up EC2 instances. Each instance installs a copy of the git repo(s), and runs a script to pull updates from production/staging branches, compiles a new build, repl…

If you make your application stateless and have it in a container then there are many managed services out there that can do this for you. For example, in AWS there is fargate and EKS.

Re: Deploys at Slack

#117

Earlier quoted context omitted.

Link doesn’t work for me right now so I haven’t read the article, but usually beta testing precedes canary deploys. Maybe this is different.

If it’s canary, you don’t trust it fully, no? Tests can pass and you still end up munging data or the user experience.

Do you ever "fully" trust any software before it's been released to its final environment and run under load? I don't.

The idea isn't that you release less-tested software because you have the canary as a safety net. The idea is that you put in place all of the other practices you would anyway to minimise the likelihood of bugs and mistakes, and then you add a canary rollout as one extra layer of protection to mitigate the damage of anything you missed.

I would look at it as 98% of the users getting an even more reliable experience than they would otherwise (per release; everyone benefits over time), rather than 2% being given a worse experience. The alternative is just that everyone is in the "canary" release and everyone has to immediately use the release you "don't fully trust".

Re: Deploys at Slack

#118

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.

Interested!

Re: Deploys at Slack

#120

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

Spinnaker does this - https://www.spinnaker.io/concepts/
Post reply on HN