Deploys at Slack • Hacker News Board

Re: Deploys at Slack

#11

Post by thinkingkong » Wed, Apr 08, 2020, 9:10 PM UTC

brycethornton wrote: post #2

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

I've built that tool 2-3 times now. The issue is really the deploy function and what controls it. It's always a one-off, or so tightly integrated into the hosting environment, that reaching in with a SaaS product is somewhat difficult. That being said, the new lowest-common-denominator standards like K8s make it way easier. If anyone is interested in using a tool just leave a comment and I'll reach out.

Re: Deploys at Slack

#12

Post by friend-monoid » Wed, Apr 08, 2020, 9:11 PM UTC

toomuchtodo wrote: post #10

friend-monoid wrote: post #8

Earlier quoted context omitted.

Seems reasonable to me? Better to deploy gradually in case the deploy is bad, right?

If the users are aware and consent to being beta testers, versus what’s already likely stable (caveat being when you’re rapidly pushing out a hotfix because your last deploy broke something).

Link doesn’t work for me right now so I haven’t read the article, but usually beta testing precedes canary deploys. Maybe this is different.

Re: Deploys at Slack

#13

Post by toomuchtodo » Wed, Apr 08, 2020, 9:12 PM UTC

friend-monoid wrote: post #12

toomuchtodo wrote: post #10

Earlier quoted context omitted.

If the users are aware and consent to being beta testers, versus what’s already likely stable (caveat being when you’re rapidly pushing out a hotfix because your last deploy broke something).

Link doesn’t work for me right now so I haven’t read the article, but usually beta testing precedes canary deploys. Maybe this is different.

If it’s canary, you don’t trust it fully, no? Tests can pass and you still end up munging data or the user experience.

Re: Deploys at Slack

#14

Post by copecopecope » Wed, Apr 08, 2020, 9:14 PM UTC

toomuchtodo wrote: post #10

friend-monoid wrote: post #8

Earlier quoted context omitted.

Seems reasonable to me? Better to deploy gradually in case the deploy is bad, right?

If the users are aware and consent to being beta testers, versus what’s already likely stable (caveat being when you’re rapidly pushing out a hotfix because your last deploy broke something).

At some point a new build needs to roll out to production. There's always going to be some risk that something goes wrong, so better to test with 2% of the population initially rather than 100%. By then, the build has already gone through integration tests/dog-fooding, so if something goes wrong in the canary phase, it's generally due to some production environment configuration issue.

Re: Deploys at Slack

#15

Post by toomuchtodo » Wed, Apr 08, 2020, 9:15 PM UTC

copecopecope wrote: post #14

toomuchtodo wrote: post #10

Earlier quoted context omitted.

If the users are aware and consent to being beta testers, versus what’s already likely stable (caveat being when you’re rapidly pushing out a hotfix because your last deploy broke something).

At some point a new build needs to roll out to production. There's always going to be some risk that something goes wrong, so better to test with 2% of the population initially rather than 100%. By then, the build has already gone through integration tests/dog-fooding, so if something goes wrong in the canary phase, it's generally due to some production environment configuration issue.

Not disagreeing, simply stating users should be aware and get a say (an option would be fine to opt in to early release access), especially if they’re a paying customer.

Re: Deploys at Slack

#16

Post by moondev » Wed, Apr 08, 2020, 9:17 PM UTC

MuffinFlavored wrote: post #4

Do they use Kubernetes at Slack?

Doesn't seem like it based on

> Instead of pushing the new build to our servers using a sync script, each server pulls the build concurrently when signaled by a Consul key change.

Re: Deploys at Slack

#17

Post by wrkronmiller » Wed, Apr 08, 2020, 9:17 PM UTC

Nice write-up! It would be interesting, however, to get more details on what types of errors were caught in dogfooding, which made it to production, what kind of hotfixes have had to be made in the past, etc...

It's nice to know what Slack does to mitigate bugs in releases, but it would also be useful to know what kinds of bugs each step catches and what bugs still slip through.

Re: Deploys at Slack

#18

Post by [deleted] » Wed, Apr 08, 2020, 9:20 PM UTC

copecopecope wrote: post #14

toomuchtodo wrote: post #10

Earlier quoted context omitted.

If the users are aware and consent to being beta testers, versus what’s already likely stable (caveat being when you’re rapidly pushing out a hotfix because your last deploy broke something).

At some point a new build needs to roll out to production. There's always going to be some risk that something goes wrong, so better to test with 2% of the population initially rather than 100%. By then, the build has already gone through integration tests/dog-fooding, so if something goes wrong in the canary phase, it's generally due to some production environment configuration issue.

[deleted]

Re: Deploys at Slack

#19

Post by copecopecope » Wed, Apr 08, 2020, 9:22 PM UTC

toomuchtodo wrote: post #15

copecopecope wrote: post #14

Earlier quoted context omitted.

At some point a new build needs to roll out to production. There's always going to be some risk that something goes wrong, so better to test with 2% of the population initially rather than 100%. By then, the build has already gone through integration tests/dog-fooding, so if something goes wrong in the canary phase, it's generally due to some production environment configuration issue.

Not disagreeing, simply stating users should be aware and get a say (an option would be fine to opt in to early release access), especially if they’re a paying customer.

I hear where you're coming from, but from my experience, the canary phase usually lasts less than an hour. And the traffic is usually split randomly, so the same 2% of users aren't at elevated risk for every deployment. I don't know how Slack does it, though.

Re: Deploys at Slack

#20

Post by derision » Wed, Apr 08, 2020, 9:23 PM UTC

capableweb wrote: post #9

No mention of feature toggles what so ever. I guess that's why it took them a long time to fix the thing with the new WYSIWYG editor, where after 2 weeks or something, they offered a toggle for people to change back. Anyone knows their reasoning behind not employing feature toggles? I would feel very slowed down if I didn't have the guarantee and confidence I could quickly rollback in the event of errors.

They had an undocumented feature toggle for that since day 1. A JavaScript snippet was issued was posted on a thread here that reverted it to the old functionality. So they are using them but not always surfacing them