Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deploys at Slack

slack.engineering

Post reply on HN
Print view View original

121–130 of 139 posts

Re: Deploys at Slack

#121

Post by alexeyindeev »

thinkingkong wrote: post #11
brycethornton wrote: post #2

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.

Interested

Re: Deploys at Slack

#122

Post by sandGorgon »

I constantly wonder if all of this UI is better expressed as a slack chat room (instead of a whole new UI)

Flowdock thought of this long time back - http://blog.flowdock.com/2014/11/11/chatops-devops-with-hubo...

Github Hubot is of course a modern interpretation of it..but I wonder why chatops doesnt have the mindshare that gitops has.

Slack's deployment is human driven. It's a natural fit for a chatops style model.

Re: Deploys at Slack

#123

Post by rickspencer3 »

Seems very relevant to many existing SaaS services today: 1. They are not doing CD, but they do deploy frequently. 2. They are using K8s, or even immutable infrastructure, so far as I can tell. 3. They have a lot of people involved in maintaining their deployments system. 4. Speaking as a user, I do not recall many significant outages, so on the surface, it seems that they have sufficient reliability.

Speaking as a heavy user of Kubernetes, evolving from an existing VM-based application to something like what Slack is doing seems like it might be more sensible than a "move everything to microservices and Kubernetes" modernization strategy.

Re: Deploys at Slack

#124

Post by thinkingkong »

sciurus wrote: post #114
thinkingkong wrote: post #11

Earlier quoted context omitted.

I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.

Please provide a way for people to reach you without commenting here.

Just ping here for now. hello@hover.sh

Re: Deploys at Slack

#125

Post by aledalgrande »

lukevp wrote: post #84
aledalgrande wrote: post #53

Earlier quoted context omitted.

My point is that if there is any downtime for the switch, for example restarting a service, it's not atomic. A small percentage of failed requests can still be high in absolute terms for a company like Slack, so why not using a paradigm [1] where you have atomic switch? And also instant rollback. [1] https://www.martinfowler.com/bliki/BlueGreenDeployment.html

Nginx can hot reload a config file while running that’s pointed at a different directory, or perhaps they’re updating a symlink?

Possible!

Re: Deploys at Slack

#126

Post by zerd »

rbtying wrote: post #61
RussianCow wrote: post #58

> Even strategies like parallel rsyncs had their limits. They don't really go into detail as to what limitations they hit by pushing code to servers instead of pulling. Does anyone have any ideas as to what those might be? I can't think of any bottlenecks that wouldn't apply in both directions, and pushing is much simpler in my experience, but I've also never been involved with deployments at this scale.

I can't speak for Slack, but it's not unreasonable to believe that a single machine's available output bandwidth (~10-40Gbps) can be saturated during a deploy of ~GB to hundreds of machines. Pushing the package to S3 and fetching it back down lets the bandwidth get spread over more machines and over different network paths (e.g. in other data centers)

We do it similarly except we push an image to a docker registry (backed by multi-region S3), then you can use e.g. ansible to pull it to 5, 10, 25, 100% of your machines. It "feels" like push though, except that you're staging the artifact somewhere. But when booting a new host it'll fetch it from the same place.

Re: Deploys at Slack

#127

Post by thanksforfish »

sandGorgon wrote: post #122

I constantly wonder if all of this UI is better expressed as a slack chat room (instead of a whole new UI) Flowdock thought of this long time back - http://blog.flowdock.com/2014/11/11/chatops-devops-with-hubo... Github Hubot is of course a modern interpretation of it..but I wonder why chatops doesnt have the mindshare that gitops has. Slack's deployment is human driven. It's a natural fit for a chatops style model.

> an engineer is designated as the deploy commander in charge of rolling out the new build to production.

When I last did ops we pushed the automation and alerting hard, so the idea of someone being formally assigned to a deployment is interesting. This sounds like they have a ton of manual or semi scripted steps. At some point, removing the dedicated deployment commander and relying on alerting is helpful, although preference of where that point is can be debated.

Re: Deploys at Slack

#128

Post by sandGorgon »

thanksforfish wrote: post #127
sandGorgon wrote: post #122

I constantly wonder if all of this UI is better expressed as a slack chat room (instead of a whole new UI) Flowdock thought of this long time back - http://blog.flowdock.com/2014/11/11/chatops-devops-with-hubo... Github Hubot is of course a modern interpretation of it..but I wonder why chatops doesnt have the mindshare that gitops has. Slack's deployment is human driven. It's a natural fit for a chatops style model.

> an engineer is designated as the deploy commander in charge of rolling out the new build to production. When I last did ops we pushed the automation and alerting hard, so the idea of someone being formally assigned to a deployment is interesting. This sounds like they have a ton of manual or semi scripted steps. At some point, removing the dedicated deployment commander and relying on alerting is helpful, although…

i think the notion of a commander is a very interesting people-ops strategy. it keeps the little element of subjectivity in things like - when do you kick off a build, how long do you run the integration/release process, etc

You do lead with automation, but the introduction of human subjectivity is a low-overhead way to still have flexibility.

Re: Deploys at Slack

#130

Post by conradfr »

icedchai wrote: post #77
echelon wrote: post #68

Earlier quoted context omitted.

> Plain EC2, backend in PHP. That's slightly horrific. Weirdware NIH deploy system, no containers, PHP.

If I was using PHP, I wouldn't use containers either. Just sync the latest code over, change a sym link to the new build, done.

That seems a little bit simplistic for today's workflow as there's chances you'll need to to restart php-fpm anyway, discard/refresh some cache (Doctrine metadada ...), maybe update your composer / vendor directory and its autoloading files, maybe run db migrations and more.
Post reply on HN