Do they use Kubernetes at Slack?
From the article it seems their deployment relies on moving application binaries to an installation directory within the VM instead of running container images.
Deploys at Slack
71–80 of 139 posts
Re: Deploys at Slack
#72It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.
I've never seen anything that could even remotely give us what we wanted. We ultimately decided to roll our own devops management platform in-house which was 100% focused on our specific needs. We are now on generation 4 of this system. We just rewrote our principal devops management interface using Blazor w/ Bootstrap4. The capabilities of the management system relative to each environment are fairly absolute - Build/Deploy/Tracing/Configuration/Reporting/etc. is all baked in. We can go from merging a PR in GitHub to a client system being updated with a fresh build of master in exactly 5 button clicks with our new system.
The central service behind the UI is a pure .NET Core solution which is responsible for executing the actual builds. The entire process is self-contained within the codebase itself. Very powerful the contract enforcement you get when the application you are building and tracking is part of the same type system as the application building and tracking it.
Re: Deploys at Slack
#73Interesting, last year I wrote a blog post on this subject and it seems pretty in line with Slack's approach :)
In regards to deployment monitoring, besides "error monitoring", I would also add "Health Monitoring" as valuable for early detection of deployment issues:
> In this line of monitoring we are interested in assuring that our application is performing as expected. First we define a set of system and business metrics that adequately represents the application behaviors. Then we start tracking these metrics, triggering an alert whenever one of them falls outside of its expected operational range. [1]
[1] https://thomasvilhena.com/2019/08/a-successful-deployment-mo...
Re: Deploys at Slack
#74Interested in how they handle DB updates/migrations (I don't know what Slack uses for data storage backend).
IMO those DB migrations are the most difficult/fraught with risk because you need to ensure that the different versions of the servers that are running as they are deploying can work with whatever state your DB is in at the moment.
Re: Deploys at Slack
#75Interested in how they handle DB updates/migrations (I don't know what Slack uses for data storage backend). IMO those DB migrations are the most difficult/fraught with risk because you need to ensure that the different versions of the servers that are running as they are deploying can work with whatever state your DB is in at the moment.
Mostly MySQL that is moving to Vitess (transparently sharded MySQL). I believe they use gh-ost for migrations.
Re: Deploys at Slack
#76I'm surprised at the 12 deployments per day, if that's truly to production. There's bugfixes etc., but feature wise Slack has been... let's say slow. Not Twitter slow, but still slow, in making any user visible changes.
Lots of places do a lot of deploys but hide significant new features behind A/B testing and feature flags. So the two things are disconnected from each other.
Re: Deploys at Slack
#77Earlier quoted context omitted.
Plain EC2, backend in PHP.
> Plain EC2, backend in PHP. That's slightly horrific. Weirdware NIH deploy system, no containers, PHP.
If I was using PHP, I wouldn't use containers either. Just sync the latest code over, change a sym link to the new build, done.
Re: Deploys at Slack
#78Earlier quoted context omitted.
Plain EC2, backend in PHP.
> Plain EC2, backend in PHP. That's slightly horrific. Weirdware NIH deploy system, no containers, PHP.
Why so? They just haven't jumped on the meme-tech train, at least wrt to this setup.
Running an actual process on an actual server has been around since time immemorial, as has doing the "atomic" deploy thing (which I'm guessing is just updating a symlink from cold to hot).
The approach is refreshingly sane.
Re: Deploys at Slack
#79Re: Deploys at Slack
#80It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.
I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.
interested