Slack’s Outage on January 4th 2021
slack.engineering
Slack’s Outage on January 4th 2021
1–10 of 110 posts
Re: Slack’s Outage on January 4th 2021
#2I was kind of surprised to see that they are using Apache's threaded workers and not nginx.
Re: Slack’s Outage on January 4th 2021
#3Wow. The trail leads back to AWS. Wasn't there a number of other companies that were down around that same time or was that a different time?
Re: Slack’s Outage on January 4th 2021
#4Wow. The trail leads back to AWS. Wasn't there a number of other companies that were down around that same time or was that a different time?
Is this the recent event you refer to? https://aws.amazon.com/message/11201/
Re: Slack’s Outage on January 4th 2021
#5I wonder why Slack uses TGW instead of VPC peering.
Re: Slack’s Outage on January 4th 2021
#6I wonder why Slack uses TGW instead of VPC peering.
The "I've just done my Solutions Architect exam" answer would be that TGW simplifies the topology by having a central hub, rather than each VPC having to peer with all the other VPCs.
I wonder how many VPCs people have before transitioning over to TGW.
Re: Slack’s Outage on January 4th 2021
#7I wonder why Slack uses TGW instead of VPC peering.
The "I've just done my Solutions Architect exam" answer would be that TGW simplifies the topology by having a central hub, rather than each VPC having to peer with all the other VPCs. I wonder how many VPCs people have before transitioning over to TGW.
I miss EC2 Classic :/. It always feels like the entire world of VPCs must have come from the armies of network engineers who felt like if the world didn't support all of the complexity they had designed to fix a problem EC2 no longer had--the tyranny of cables and hubs and devices acting as routers--that maybe they would be out of a job or something, and so rather than design hierarchical security groups Amazon just brought back in every feature of network administration I had been happily prepared to never have to think about every again :(.
Re: Slack’s Outage on January 4th 2021
#8I was kind of surprised to see that they are using Apache's threaded workers and not nginx.
Apache mod_php is still much faster than php-fpm, and since slack uses a lot of PHP on the backend it makes a lot of sense for them.
Re: Slack’s Outage on January 4th 2021
#9> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on their first connection to Slack. We go from our quietest time of the whole year to one of our biggest days quite literally overnight.
What's interesting is that when this happened, some HN comments suggested it was the return from holiday traffic that caused it. Others said, "nah, don't you think they know how to handle that by now?"
Turns out occam's razor applied here. The simplest answer was the correct one. Return-from-holiday traffic.
Re: Slack’s Outage on January 4th 2021
#10I really enjoy reading these write-ups, even if the causal incident is not something I enjoy.