Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deploys at Slack

slack.engineering

Post reply on HN
Print view View original

61–70 of 139 posts

Re: Deploys at Slack

#61

Post by rbtying »

RussianCow wrote: post #58

> Even strategies like parallel rsyncs had their limits. They don't really go into detail as to what limitations they hit by pushing code to servers instead of pulling. Does anyone have any ideas as to what those might be? I can't think of any bottlenecks that wouldn't apply in both directions, and pushing is much simpler in my experience, but I've also never been involved with deployments at this scale.

I can't speak for Slack, but it's not unreasonable to believe that a single machine's available output bandwidth (~10-40Gbps) can be saturated during a deploy of ~GB to hundreds of machines. Pushing the package to S3 and fetching it back down lets the bandwidth get spread over more machines and over different network paths (e.g. in other data centers)

Re: Deploys at Slack

#62

Post by taleodor »

brycethornton wrote: post #2

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

This is part of what we're doing with Reliza Hub - https://relizahub.com (note, we're in a very early stage).

Apart from tracking deployments, we're really focused on tracking bills of materials and communication between Business and Tech teams.

Re: Deploys at Slack

#63

Post by [deleted] »

nathankunicki wrote: post #37

Fun to read, but there's a lack of detail here that I'd like to see. For example, this talks purely about code changes. However times a code change requires a database schema change (as mentioned above), different API's to be used, etc. In the percentage based rollout where multiple versions are in use at once, how are these differences handled?

[deleted]

Re: Deploys at Slack

#64

Post by one2know »

RussianCow wrote: post #58

> Even strategies like parallel rsyncs had their limits. They don't really go into detail as to what limitations they hit by pushing code to servers instead of pulling. Does anyone have any ideas as to what those might be? I can't think of any bottlenecks that wouldn't apply in both directions, and pushing is much simpler in my experience, but I've also never been involved with deployments at this scale.

Considering they are not bringing machines out of rotation or draining connections in the example given with the errors, I assume that more than 10 machines produces too many errors or takes too long to have two versions of the code deployed, and wherever they pull from is not scalable. All those problems can be easily solved though.

Re: Deploys at Slack

#65

Post by Saaster »

MuffinFlavored wrote: post #36
moondev wrote: post #16

Earlier quoted context omitted.

Doesn't seem like it based on > Instead of pushing the new build to our servers using a sync script, each server pulls the build concurrently when signaled by a Consul key change.

does that mean they are not even using containers?

Plain EC2, backend in PHP.

Re: Deploys at Slack

#66

Post by Saaster »

I'm surprised at the 12 deployments per day, if that's truly to production. There's bugfixes etc., but feature wise Slack has been... let's say slow. Not Twitter slow, but still slow, in making any user visible changes.

Re: Deploys at Slack

#67

Post by Silhouette »

codenesium wrote: post #57
Silhouette wrote: post #52

It's interesting that atomic deploys weren't in from the start. That was one of the few deployment practices we really insisted on from day one at my own businesses, if only because the uncertainty you get from trying to trace problems where your system isn't in any known state makes it all but impossible to work systematically. A related challenge where we've never really found a good solution is how to handle deplo…

I've never seen it solved. You either write and test migration scripts to roll it back or you restore from a backup. Idk what you do if you add a new column that's populated in the new version and you rollback. I guess this would be a good place to roll out as small of piece as you can and hope you don't find out it's busted a week later.

Idk what you do if you add a new column that's populated in the new version and you rollback.

I think that one depends on what you are rolling back and whether you have your application code somewhat isolated from your underlying database via a well-defined API.

Assuming that you will at some point need to populate your new column for all your pre-existing records in some well-defined way, you can handle rolling back the application code as long as you have a version of the database API that still provides the interface the older application code requires. You might no longer be updating your new column with new data at that time, but the data you did get is still there, and when you later want to move your application code forward again you can populate the new column for any extra records that have been added to your database in the meantime just as you did on the initial migration.

Given the practicalities of a multi-step migration involving both application and database schema, you might already have the necessary extra code in your database API to support running old application code against the new database schema, and even to fill in any missing data for that extra column according to the same rules you used for migrating older data from before transition and ensure any new constraints are satisfied. So this way, you can wind back your application code but not damage your new database.

If for some reason the database schema itself needs to be rolled back, and you can't just fake it at the API level, things become a lot more difficult as you have potential data loss issues to contend with. Likewise if it's possible that the old application code would not maintain any new database records in a way that satisfies all required constraints and you can't handle that at the API. Fortunately, this doesn't seem to happen very often in practice.

Re: Deploys at Slack

#70

Post by lukax »

mrdonbrown wrote: post #27
brycethornton wrote: post #2

It's always nice to see how other teams do it. Nothing too groundbreaking here but that's a good thing. I did notice the screenshot of "Checkpoint", their deployment tracking UI. Are there solid open source or SaaS tools doing something similar? I've seen various companies build similar tools but most deployment processes are consistent enough to have a 3rd-party tool that was useful for most teams.

Sleuth is a SaaS deployment tracker that pulls deployments from source repositories, feature flags, and other sources, in addition to pushes via curl. You can see Sleuth used to, well, track Sleuth at https://app.sleuth.io/sleuth [Disclaimer: am a Sleuth co-founder]

I can also recommend Sleuth. We use it at our company and the integration is very good. Their team is constantly working on new features, integrations and better UI.

Hi Don :)

Post reply on HN