Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deploys at Slack

slack.engineering

Post reply on HN
Print view View original

131–139 of 139 posts

Re: Deploys at Slack

#131

Post by antpls »

sandGorgon wrote: post #128
thanksforfish wrote: post #127

Earlier quoted context omitted.

> an engineer is designated as the deploy commander in charge of rolling out the new build to production. When I last did ops we pushed the automation and alerting hard, so the idea of someone being formally assigned to a deployment is interesting. This sounds like they have a ton of manual or semi scripted steps. At some point, removing the dedicated deployment commander and relying on alerting is helpful, although…

i think the notion of a commander is a very interesting people-ops strategy. it keeps the little element of subjectivity in things like - when do you kick off a build, how long do you run the integration/release process, etc You do lead with automation, but the introduction of human subjectivity is a low-overhead way to still have flexibility.

More likely, they have a few bugs here and there in the deployment tools that require human supervision and intervention, and they don't have the resources right now to fix them and to make it more reliable.

There is no need for flexibility in a repetitive process, unless there are bugged edge cases

Re: Deploys at Slack

#132

Post by switch007 »

hn_throwaway_99 wrote: post #74

Interested in how they handle DB updates/migrations (I don't know what Slack uses for data storage backend). IMO those DB migrations are the most difficult/fraught with risk because you need to ensure that the different versions of the servers that are running as they are deploying can work with whatever state your DB is in at the moment.

It’s almost a running joke that if a big or well known company blogs about their deploys, they won’t go into detail about databases.

Re: Deploys at Slack

#133

Post by mleonhard »

12 deploys in an 8-hour window is only 40 min per deploy. Do they really perform all of those steps in 40 mins, or do they have multiple deployments going at once (pipelining)?

Re: Deploys at Slack

#134

Post by mleonhard »

How do they choose which shards are included in the first 10% canary group?

This is a tricky problem. It's tempting to include only small (less valuable) accounts in the first group. But some bugs only occur with large accounts, so you need some of those in the first 10%.

Many bugs affect only a small portion of customers. There are many categories. A canary becomes more effective when it includes members from each category. Example: account type, number of users, client type (web/ios/android/macos/windows/linux), client version, web browser type and version, ipv4/ipv6, vpn, TLS MITM proxy, language, timezone, payment currency, country, tax region, mobile service provider, etc.

Re: Deploys at Slack

#135

Post by roadbeats »

gtsteve wrote: post #103
roadbeats wrote: post #99

It's cool to see how big organizations have deployment setups, while it feels like there is not enough resources about how one should setup a deployment system for a new startup just in the beginning. The setup I currently use is custom bash scripts setting up EC2 instances. Each instance installs a copy of the git repo(s), and runs a script to pull updates from production/staging branches, compiles a new build, repl…

You don't have to do a big-bang style Google thing. You can just invest in some continuous improvement over the next few years: Iteration 0: What you have now. Iteration 1: A build server builds your artifact, and your EC2 instances download the artifact from the build server. Iteration 2: The build server builds the artifact and builds a container and pushes it to ECR. Your EC2 instances now pull the image into Dock…

Thanks a lot for the answer.

Re: Deploys at Slack

#136

Post by roadbeats »

diamondo25 wrote: post #102
roadbeats wrote: post #99

It's cool to see how big organizations have deployment setups, while it feels like there is not enough resources about how one should setup a deployment system for a new startup just in the beginning. The setup I currently use is custom bash scripts setting up EC2 instances. Each instance installs a copy of the git repo(s), and runs a script to pull updates from production/staging branches, compiles a new build, repl…

Automate compilation on a buildserver and run tests on that, and if everything is ok, use the artifacts to push to your servers. This way you can guarantee that the code is tested and all running versions are from the same build environment.

Thanks for the answer.

Re: Deploys at Slack

#137

Post by elbear »

sandGorgon wrote: post #119
elbear wrote: post #108

Earlier quoted context omitted.

For Kubernetes, there's this: https://github.com/lensapp/lens

This is super cool. I wonder if there's anything like this as a vscode plugin

Someone mentioned this in another thread: https://github.com/GoogleCloudPlatform/cloud-code-vscode

Re: Deploys at Slack

#138

Post by GordonS »

roadbeats wrote: post #99

It's cool to see how big organizations have deployment setups, while it feels like there is not enough resources about how one should setup a deployment system for a new startup just in the beginning. The setup I currently use is custom bash scripts setting up EC2 instances. Each instance installs a copy of the git repo(s), and runs a script to pull updates from production/staging branches, compiles a new build, repl…

For me, it feels a bit "wrong" to be building on each production server.

Firstly, production servers are usually "hardened", and only have installed what they need to run, reducing the attack surface as much as possible.

Secondly, for proprietary code, I don't want it on production servers.

But most importantly, I want a single, consistent set of build artifacts that can be deployed across the server/container fleet.

You can do this with CI/CD tools, such as Azure DevOps (my personal favourite), Github Actions, CircleCI, Jenkins and Appveyor.

The way it works is you set up an automated build pipeline, so when you push new code, it's built once, centrally, and the build output is made available as "build artifacts". In another pipeline stage, you can then push out the artifacts to your servers using various means (rsync, FTP, build agent, whatever), or publish them somewhere (S3, Docker Registry, whatever) where your servers can pull them from. You can have more advanced workflows, but that's the basic version.

Re: Deploys at Slack

#139

Post by GordonS »

mleonhard wrote: post #133

12 deploys in an 8-hour window is only 40 min per deploy. Do they really perform all of those steps in 40 mins, or do they have multiple deployments going at once (pipelining)?

I was thinking the same, especially since they mention manual testing.
Post reply on HN