Slack’s Outage on January 4th 2021 • Hacker News Board

Re: Slack’s Outage on January 4th 2021

#11

Post by saurik » Mon, Feb 01, 2021, 2:32 PM UTC

dijit wrote: post #8

fullstop wrote: post #2

I was kind of surprised to see that they are using Apache's threaded workers and not nginx.

Apache mod_php is still much faster than php-fpm, and since slack uses a lot of PHP on the backend it makes a lot of sense for them.

I loved this article where someone started with the goal of writing an article about how much faster nginx was, but then discovered the opposite (and for some reason didn't change the title of the article, which is hilarious)... both because it showed the author "cared", but also because it showed that people just assume what amounts to marketing myths (such as that Apache and mod_php are ancient tech vs. the more modern php-fpm stack) before bothering to verify anything.

https://www.eschrade.com/page/why-is-fastcgi-w-nginx-so-much...

Re: Slack’s Outage on January 4th 2021

#12

Post by Denvercoder9 » Mon, Feb 01, 2021, 2:32 PM UTC

dijit wrote: post #8

fullstop wrote: post #2

I was kind of surprised to see that they are using Apache's threaded workers and not nginx.

Apache mod_php is still much faster than php-fpm, and since slack uses a lot of PHP on the backend it makes a lot of sense for them.

That's not the reason, as mod_php only supports the prefork MPM, and not the threaded MPMs.

Re: Slack’s Outage on January 4th 2021

#13

Post by cett » Mon, Feb 01, 2021, 2:40 PM UTC

dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

Though the nuance is Slack did know how to handle it, AWS didn't.

Re: Slack’s Outage on January 4th 2021

#14

Post by tempest_ » Mon, Feb 01, 2021, 2:43 PM UTC

cett wrote: post #13

dplgk wrote: post #9

> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…

Though the nuance is Slack did know how to handle it, AWS didn't.

Well Slack depended on the Cloud(tm).

It is a interesting though because a lot of the blog posts like "How we handled a 3000% traffic increase overnight!" boil down to "We turned up the AWS knob".

What happens when the AWS knob doesn't work?

Re: Slack’s Outage on January 4th 2021

#15

Post by gscho » Mon, Feb 01, 2021, 2:51 PM UTC

> our dashboarding and alerting service became unavailable.

Sounds like the monitoring system needs a monitoring system.

Re: Slack’s Outage on January 4th 2021

#16

Post by johannes1234321 » Mon, Feb 01, 2021, 2:54 PM UTC

Denvercoder9 wrote: post #12

dijit wrote: post #8

Earlier quoted context omitted.

Apache mod_php is still much faster than php-fpm, and since slack uses a lot of PHP on the backend it makes a lot of sense for them.

That's not the reason, as mod_php only supports the prefork MPM, and not the threaded MPMs.

Threaded MPMs and PHP work quite well, I fixed a few bugs in that space some time ago.

There however might be issues in some of the millions libraries PHP potentially links in an called from it's extensions, and those sometimes at emit thread safe, but finding that and finding bypasses isn't easy ...

The other issue is that it's often slower (while there recently were changes from custom thread local storage to more modern one) and if there is a crash (i.e. Recursion stack overflow ...) it affects all requests in that process, not only the one.

Re: Slack’s Outage on January 4th 2021

#17

Post by BillinghamJ » Mon, Feb 01, 2021, 2:55 PM UTC

saurik wrote: post #7

tikkabhuna wrote: post #6

Earlier quoted context omitted.

The "I've just done my Solutions Architect exam" answer would be that TGW simplifies the topology by having a central hub, rather than each VPC having to peer with all the other VPCs. I wonder how many VPCs people have before transitioning over to TGW.

I miss EC2 Classic :/. It always feels like the entire world of VPCs must have come from the armies of network engineers who felt like if the world didn't support all of the complexity they had designed to fix a problem EC2 no longer had--the tyranny of cables and hubs and devices acting as routers--that maybe they would be out of a job or something, and so rather than design hierarchical security groups Amazon just…

Generally inclined to agree, but to be fair you can operate a VPC in exactly the same way as EC2 Classic - give everything public IPs, public subnets and ignore the internal IPs. Pretty sure those are the defaults too

Re: Slack’s Outage on January 4th 2021

#18

Post by conradfr » Mon, Feb 01, 2021, 2:57 PM UTC

nickthemagicman wrote: post #3

Wow. The trail leads back to AWS. Wasn't there a number of other companies that were down around that same time or was that a different time?

Does AWS compensate you in cases like this?

Re: Slack’s Outage on January 4th 2021

#19

Post by whatisthiseven » Mon, Feb 01, 2021, 2:58 PM UTC

saurik wrote: post #7

tikkabhuna wrote: post #6

Earlier quoted context omitted.

The "I've just done my Solutions Architect exam" answer would be that TGW simplifies the topology by having a central hub, rather than each VPC having to peer with all the other VPCs. I wonder how many VPCs people have before transitioning over to TGW.

I miss EC2 Classic :/. It always feels like the entire world of VPCs must have come from the armies of network engineers who felt like if the world didn't support all of the complexity they had designed to fix a problem EC2 no longer had--the tyranny of cables and hubs and devices acting as routers--that maybe they would be out of a job or something, and so rather than design hierarchical security groups Amazon just…

Agreed, I always thought VPC and all that complexity was a big step backwards. My org is moving from a largely managed network into AWS, and now we have to configure the whole network and external gateways ourselves? What engineer wants to do this?

VPCs are virtual, but I don't need VPCs, I need the entire network layer virtualized and abstracted. As you suggested,just grouping devices in a single network and saying "let them all talk to each other, let this one talk to that one over this port/IP" should be all I describe. Let AWS figure out CIDR, routing, gateways, etc.

Re: Slack’s Outage on January 4th 2021

#20

Post by fullstop » Mon, Feb 01, 2021, 3:01 PM UTC

saurik wrote: post #11

dijit wrote: post #8

Earlier quoted context omitted.

Apache mod_php is still much faster than php-fpm, and since slack uses a lot of PHP on the backend it makes a lot of sense for them.

I loved this article where someone started with the goal of writing an article about how much faster nginx was, but then discovered the opposite (and for some reason didn't change the title of the article, which is hilarious)... both because it showed the author "cared", but also because it showed that people just assume what amounts to marketing myths (such as that Apache and mod_php are ancient tech vs. the more mo…

When I moved things from Apache -> nginx years ago, I did it not because it was faster but because the resource requirements of nginx were so much more predictable under load.