Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Outage on January 4th 2021

slack.engineering

Post reply on HN
Print view View original

31–40 of 110 posts

Re: Slack’s Outage on January 4th 2021

#31

Post by oxfordmale »

cett wrote: post #13
dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

Though the nuance is Slack did know how to handle it, AWS didn't.

If you hit yourself on the thumb while using a hammer, do you blame the hammer manufacturer or yourself? TGW limits are well documented.

Re: Slack’s Outage on January 4th 2021

#32

Post by dewey »

conradfr wrote: post #18
nickthemagicman wrote: post #3

Wow. The trail leads back to AWS. Wasn't there a number of other companies that were down around that same time or was that a different time?

Does AWS compensate you in cases like this?

It depends: https://aws.amazon.com/legal/service-level-agreements/

Re: Slack’s Outage on January 4th 2021

#33

Post by thunderbong »

dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

But Slack has been around for longer than a year, right? Shouldn't they have noticed this happening earlier?

I mean, considering Slack is mostly used as a workplace chat mechanism, they should have faced this kind of a scenario previously and had a solution for this by now.

Re: Slack’s Outage on January 4th 2021

#34

Post by kparaju »

Some lessons I took from this retro:

- Disable autoscaling if appropriate during outage. For example if the web server is degraded, it's probably best to make sure that the backends don't autoscale down.

- Panic mode in Envoy is amazing!

- Ability to quickly scale your services is important, but that metric should also take into account how quickly the underlying infrastructure can scale. Your pods could spin up in 15 seconds but k8s nodes will not!

Re: Slack’s Outage on January 4th 2021

#36

Post by 0dmethz »

oxfordmale wrote: post #31
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

If you hit yourself on the thumb while using a hammer, do you blame the hammer manufacturer or yourself? TGW limits are well documented.

I mean, when the hammer manufacturer sells managed, auto scaling thumb-avoiding services you might rely on that.

If I understand correctly they didn't initially hit a TGW quota, it just didn't scale up fast enough.

Re: Slack’s Outage on January 4th 2021

#37

Post by delfaras »

thunderbong wrote: post #33
dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

But Slack has been around for longer than a year, right? Shouldn't they have noticed this happening earlier? I mean, considering Slack is mostly used as a workplace chat mechanism, they should have faced this kind of a scenario previously and had a solution for this by now.

Yeah but this year the number of people working from home that would connect to slack directly at the beginning of their work day must be much much larger than the other years

Re: Slack’s Outage on January 4th 2021

#38

Post by tw04 »

fipar wrote: post #27
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck.

Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling?

If you're saying now the story is: well rely on them to auto scale, until they don't - then why would I bother? Now you're telling me I need to go back to having infrastructure experts, which means I can save TON of money by going with a hosting provider that allows allocation of resources via API (which is basically all of them).

Re: Slack’s Outage on January 4th 2021

#39

Post by twblalock »

cett wrote: post #13
dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

Though the nuance is Slack did know how to handle it, AWS didn't.

Lots of other services that use AWS didn't go down the same day -- because they provisioned enough AWS capacity.

Re: Slack’s Outage on January 4th 2021

#40

Post by helper »

twblalock wrote: post #39
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

Lots of other services that use AWS didn't go down the same day -- because they provisioned enough AWS capacity.

Where's the button to provision more TGW capacity?
Post reply on HN