Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Outage on January 4th 2021

slack.engineering

Post reply on HN
Print view View original

41–50 of 110 posts

Re: Slack’s Outage on January 4th 2021

#41

Post by twblalock »

tw04 wrote: post #38
fipar wrote: post #27

Earlier quoted context omitted.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…

If you are serious about reliability you always need infrastructure experts.

AWS is pretty good about documenting the limits of their systems, SLAs, how to configure them, etc. They don't just say you should wave a magic wand -- and even if they did say that, professional software engineers know better.

"a hosting provider that allows allocation of resources via API" is exactly what AWS is. Your infrastructure experts come into the picture because they need to know which resources to request, how to estimate the scale they need, and how to configure them properly. They should also be doing performance testing to see if the claimed performance really holds up.

Re: Slack’s Outage on January 4th 2021

#42

Post by ncallaway »

tw04 wrote: post #38
fipar wrote: post #27

Earlier quoted context omitted.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…

> Isn't that literally supposed to be the sales pitch for the cloud?

Yes.

But a sales pitch is the most positive framing of the product possible. I wouldn't rely on the sales pitch when making the decision about how much you should depend on the cloud.

Re: Slack’s Outage on January 4th 2021

#43

Post by remus »

tw04 wrote: post #38
fipar wrote: post #27

Earlier quoted context omitted.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…

> Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling?

Clearly there are limits even with the largest cloud providers. You'll have to engage a bit of critical thought in to whether you're going to get near those limits and what that might mean for your product. Obviously that's easier said than done, but you could argue that the cloud providers are still giving you reasonable value if you can pass the buck on a given issue for x years.

Re: Slack’s Outage on January 4th 2021

#44

Post by solidasparagus »

tw04 wrote: post #38
fipar wrote: post #27

Earlier quoted context omitted.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…

No, the cloud provides scalable infrastructure, but once you are in the 0.01% and you have very unique usage patterns, you still need to know how to set up your infrastructure for your needs. The difference is that instead of writing and managing a scalable cache, you just need to build the layer that knows to pre-provision for that scale/talk with AWS to make sure the system has sufficient capacity.

The cloud isn't some magic thing that solves all scaling problems, it's a tool that gives you strong primitives (and once you're a large enough customer, an active partner) to help you solve your scaling problems.

Re: Slack’s Outage on January 4th 2021

#45

Post by jabart »

cett wrote: post #13
dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

Though the nuance is Slack did know how to handle it, AWS didn't.

If you have a known increase in traffic at a certain date/time that auto-scaling (either EC2 or NLB/ALB or another service) can't handle fast enough you can let AWS know through your support contract to over-provision during that time or the scale up will take too long.

Re: Slack’s Outage on January 4th 2021

#46

Post by cholmon »

dijit wrote: post #8
fullstop wrote: post #2

I was kind of surprised to see that they are using Apache's threaded workers and not nginx.

Apache mod_php is still much faster than php-fpm, and since slack uses a lot of PHP on the backend it makes a lot of sense for them.

Are you saying that Slack uses mod_php? According to a Slack Engineering blog post[0] from 9 months ago, Slack has been using Hack/HHVM since 2016 in place of PHP. My understanding is that HHVM can only be run over FastCGI, unless there's a mod_hhvm that I'm unaware of.

[0]: https://slack.engineering/hacklang-at-slack-a-better-php/

Re: Slack’s Outage on January 4th 2021

#47

Post by miyuru »

Thaxll wrote: post #30

Surprised that there is just a few metrics available for TGW: https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-c... Probably the only way to see a problem is if you have a flat line for bandwidth, but as the article suggested they had packet drop wich does not appear on the cloudwatch metrics, aws should add those metrics imo

some TGW limits are documented here: https://aws.amazon.com/transit-gateway/faqs/#Performance_and...

Re: Slack’s Outage on January 4th 2021

#48

Post by steve_adams_86 »

delfaras wrote: post #37
thunderbong wrote: post #33

Earlier quoted context omitted.

But Slack has been around for longer than a year, right? Shouldn't they have noticed this happening earlier? I mean, considering Slack is mostly used as a workplace chat mechanism, they should have faced this kind of a scenario previously and had a solution for this by now.

Yeah but this year the number of people working from home that would connect to slack directly at the beginning of their work day must be much much larger than the other years

Another small but potentially relevant detail: Not many people vacationed, so more people would have returned to work at standard times. Many people travel over holidays for example (usually) but this time around in many places it wasn't even an option. Other people extend their holidays to relax more, but I don't know any people interested in staycations in their house. We've had enough of it.

Re: Slack’s Outage on January 4th 2021

#50

Post by jeffrallen »

tldr: "we added complexity into our system to make it safer and that complexity blew us up. Also: we cannot scale up our system without everything working well, so we couldn't fix our stuff. Also: we were flying blind, probably because of the failing complexity that was supposed to protect us."

I am really not impressed... with the state of IT. I could not have done better, but isn't it too bad that we've built these towers of sand that keep knocking each other over?

Post reply on HN