Can anyone explain why they do 12 deploys a day? Are engineers pushing to production as a way of iteratively testing a feature?
Deploys at Slack
101–110 of 139 posts
Re: Deploys at Slack
#102It's cool to see how big organizations have deployment setups, while it feels like there is not enough resources about how one should setup a deployment system for a new startup just in the beginning. The setup I currently use is custom bash scripts setting up EC2 instances. Each instance installs a copy of the git repo(s), and runs a script to pull updates from production/staging branches, compiles a new build, repl…
Automate compilation on a buildserver and run tests on that, and if everything is ok, use the artifacts to push to your servers. This way you can guarantee that the code is tested and all running versions are from the same build environment.
Re: Deploys at Slack
#103It's cool to see how big organizations have deployment setups, while it feels like there is not enough resources about how one should setup a deployment system for a new startup just in the beginning. The setup I currently use is custom bash scripts setting up EC2 instances. Each instance installs a copy of the git repo(s), and runs a script to pull updates from production/staging branches, compiles a new build, repl…
You don't have to do a big-bang style Google thing. You can just invest in some continuous improvement over the next few years:
Iteration 0: What you have now.
Iteration 1: A build server builds your artifact, and your EC2 instances download the artifact from the build server.
Iteration 2: The build server builds the artifact and builds a container and pushes it to ECR. Your EC2 instances now pull the image into Docker and start it.
Iteration 3: You use ECS for basic container orchestration. Your build server instructs your ECS instances to download the image and run them, with blue-green deployments linked to your load balancer.
Iteration 4: You set up K8s and your build server instructs it to deploy.
I went in a similar trajectory, and I'm at iteration 3 right now, on the verge of moving to K8s.
It's your call on how long the timespan is here, and commercial pressures will drive it. It could be 6 months, it could be 3 years.
Re: Deploys at Slack
#104It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.
I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.
Interested!
Re: Deploys at Slack
#105I'm surprised at the 12 deployments per day, if that's truly to production. There's bugfixes etc., but feature wise Slack has been... let's say slow. Not Twitter slow, but still slow, in making any user visible changes.
Far too many people on HN seem to think the public facing code that we see is all that the engineering team in a large company works on. There's so much more to running a large SaaS business. If Slack is like all the other SaaS companies I've encountered they'll have dozens of internal apps for sales, comms, analytics, engineering, etc that they work on that people outside of the business never see[1]. Those all need developing and all need deploying.
[1] They might buy in solutions for some business functions like accounting, HR and support, but they'll still have tons of homegrown stuff. Every tech company does.
Re: Deploys at Slack
#106Can anyone explain why they do 12 deploys a day? Are engineers pushing to production as a way of iteratively testing a feature?
Multiple teams working on different features rolling their local commits into release branches - and rather than feature related deploys they do time-boxed ones by the looks of it so that they have dedicated support on hand to spot issues and rollback immediately (guessing from what I can see based on the screenshot).
Easy to get that many deploys out the door if you have a managed process like this - fast iteration, lots of different feature bumps and tweaks, different locales, updating even 1/2 links or words in a hardcoded page...
Re: Deploys at Slack
#107Earlier quoted context omitted.
> Plain EC2, backend in PHP. That's slightly horrific. Weirdware NIH deploy system, no containers, PHP.
If I was using PHP, I wouldn't use containers either. Just sync the latest code over, change a sym link to the new build, done.
capistrano is perfect for that. we use it for all our deployment needs and it has been wonderful!
Re: Deploys at Slack
#108Earlier quoted context omitted.
I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.
Interested, especially in K8s based
For Kubernetes, there's this: https://github.com/lensapp/lens
Re: Deploys at Slack
#109I wonder why they didn't evaluate at some point using an immutable infrastructure approach leveraging tools like Spinnaker to manage the deploy? They sure have the muscle and numbers to use it and even contribute to it actively, no? I mean, I know that deploying your software is usually something pretty tied to a specific engineering team but I really like the immutable approach and I was wondering why a company the…
I had similar thoughts when I read their article. Their atomic deploy problem completely disappears had they gone with an immutable approach.
Re: Deploys at Slack
#110Can anyone explain why they do 12 deploys a day? Are engineers pushing to production as a way of iteratively testing a feature?
They're not deploying untested software, if that's what you're asking. They most likely simply deploy each change when it is ready, rather than building up work-in-progress and deploying many changes at the same time. It's a lot safer to change one thing at a time, see https://www.goodreads.com/en/book/show/35747076-accelerate. Releasing changes as soon as they are ready can also enable them to gather feedback faster - in this sense they would be iteratively 'testing' the product.