I wonder why Slack uses TGW instead of VPC peering.
this article from 4 months ago explains it.
https://slack.engineering/building-the-next-evolution-of-clo...
Slack’s Outage on January 4th 2021
21–30 of 110 posts
Re: Slack’s Outage on January 4th 2021
#22> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…
Though the nuance is Slack did know how to handle it, AWS didn't.
I actually thought something in AWS was a cause but did not know about these internal systems.
Re: Slack’s Outage on January 4th 2021
#23> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…
Though the nuance is Slack did know how to handle it, AWS didn't.
I actually thought something in AWS was a cause but did not know anything about how TGW works internally.
Re: Slack’s Outage on January 4th 2021
#24> our dashboarding and alerting service became unavailable. Sounds like the monitoring system needs a monitoring system.
"Who monitors the monitors?"
Re: Slack’s Outage on January 4th 2021
#25I wonder why Slack uses TGW instead of VPC peering.
The "I've just done my Solutions Architect exam" answer would be that TGW simplifies the topology by having a central hub, rather than each VPC having to peer with all the other VPCs. I wonder how many VPCs people have before transitioning over to TGW.
Exams these days are a cross between pre-sales training and AWS dogma. Fundamentally all AWS services share the same primitives. It would be great if AWS could take incidents like these, provide some guidance on how to avoid them, and then add 1 or 2 questions to the exam. It would give some credence to the exams which are now basically crammed material.
Re: Slack’s Outage on January 4th 2021
#26I wonder if you can pre-warm TGWs like you can ELB? It would be annoying to have to have AWS prewarm a bunch of you stuff, but it's better than it going down.
Re: Slack’s Outage on January 4th 2021
#27> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…
Though the nuance is Slack did know how to handle it, AWS didn't.
I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck.
This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too.
All this to say that the cloud isn't magic. From a risk/error prevention point of view, it's not that different from writing software for a single local machine: not every programmer needs to know how to manually do memory management, it makes a lot more sense to rely on your OS and malloc (and friends) for this, but the caveat is that you do need to account for the fact that malloc may fail. In the cloud case, one can't just assume that you'll always be able to provision a new instance, scale up a service, etc. The cloud is like a utility company: normally very reliable, but they do fail too.
Re: Slack’s Outage on January 4th 2021
#28> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…
But what HN predicted was wrong https://news.ycombinator.com/item?id=25632346
"My bet is that this incident is caused by a big release after a post-holiday "code freeze". "
Re: Slack’s Outage on January 4th 2021
#29Earlier quoted context omitted.
The "I've just done my Solutions Architect exam" answer would be that TGW simplifies the topology by having a central hub, rather than each VPC having to peer with all the other VPCs. I wonder how many VPCs people have before transitioning over to TGW.
I miss EC2 Classic :/. It always feels like the entire world of VPCs must have come from the armies of network engineers who felt like if the world didn't support all of the complexity they had designed to fix a problem EC2 no longer had--the tyranny of cables and hubs and devices acting as routers--that maybe they would be out of a job or something, and so rather than design hierarchical security groups Amazon just…
My assumption is it's an IPv4 address exhaustion thing too.
Re: Slack’s Outage on January 4th 2021
#30Surprised that there is just a few metrics available for TGW: https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-c...
Probably the only way to see a problem is if you have a flat line for bandwidth, but as the article suggested they had packet drop wich does not appear on the cloudwatch metrics, aws should add those metrics imo