Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack’s Outage on January 4th 2021

slack.engineering

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Slack’s Outage on January 4th 2021

#81

Post by bryan_w »

rrrrrrrrrrrryan wrote: post #74
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

Some of my co-workers came from active.com (a website that lets people register for marathons and events). The infrastructure had to handle massive spikes because registrations for big races would open all at once, so scalability was everything. They explained to me that they'd intentionally slam the production website with external traffic a couple of times per year, at a scheduled time in the middle of the night. L…

The problem you run into is that while you can load test your website with no problems, when running on shared infrastructure (AWS), you have to account for everyone's website being under load at the same time. That isn't as easy to test or find bottlenecks for.

Re: Slack’s Outage on January 4th 2021

#82

Post by user5994461 »

dijit wrote: post #8
fullstop wrote: post #2

I was kind of surprised to see that they are using Apache's threaded workers and not nginx.

Apache mod_php is still much faster than php-fpm, and since slack uses a lot of PHP on the backend it makes a lot of sense for them.

mod_xyz plugins were deprecated long ago because they are unstable. If you do a comparison you should compare fastcgi versus fastcgi, that is the standard way to run web applications. Running with mod should be faster because it's running the interpreter directly into the apache process but it's also making apache unstable.

mod_python was abandoned around a decade ago. It's crashing on python 2.7.

mod_perl was dropped in 2012 with the release of apache 2.4. It was kicked out of the project but continues to exist as a separate project (not sure if it works at all).

Re: Slack’s Outage on January 4th 2021

#83

Post by jen20 »

Lammy wrote: post #29
saurik wrote: post #7

Earlier quoted context omitted.

I miss EC2 Classic :/. It always feels like the entire world of VPCs must have come from the armies of network engineers who felt like if the world didn't support all of the complexity they had designed to fix a problem EC2 no longer had--the tyranny of cables and hubs and devices acting as routers--that maybe they would be out of a job or something, and so rather than design hierarchical security groups Amazon just…

My assumption is it's an IPv4 address exhaustion thing too.

It's more to do with what entries you would put in a routing table to get EC2 Classic over a DirectConnect, no?

Re: Slack’s Outage on January 4th 2021

#84

Post by floatingatoll »

nicoburns wrote: post #73
floatingatoll wrote: post #66

Earlier quoted context omitted.

If your oldest request was queued 5+ seconds ago in a near-realtime system (such as Slack), CPU usage isn't your biggest problem. Slack wrote an autoscaling implementation that ignored request queue depth and downsized their cluster based on CPU usage alone, so while they knew how to resolve it, I would not go so far as to say they knew how to prevent it. The mistake of ignoring the maxage of the request queue is per…

> The mistake of ignoring the maxage of the request queue is perhaps the second most common blind spot in every Ops team I've ever worked with. No insult to my fellow Ops folks, but we've got to stop overlooking this. What's the first?

Non-randomized wallclock integers.

For example: “sleep 60 seconds”, “cron 0 * * * * command”, “X-Retry-After: 300”

Found in: recurring jobs, backoff algorithms, oauth tokens.

Found in: ops-created tasks, dev-released software.

Re: Slack’s Outage on January 4th 2021

#85

Post by [deleted] »

bengale wrote: post #78
jrockway wrote: post #62

The thing that always worries me about cloud systems are the hidden dependencies in your cloud provider that work until they don't. They typically don't output logs and metrics, so you have no choice to pray that someone looks at your support ticket and clicks their internal system's "fix it for this customer" button. I'll also say that I'm interested in ubiquitous mTLS so that you don't have to isolate teams with VP…

Experienced something similar with mongo atlas today. Our primary node went down and the cluster didn’t failover to either of the secondaries. We got to sit with our production environment completely offline while staring at two completely functional nodes that we had no ability to use. Even when we managed to get hold of support they also seemed unable to trigger a failover and basically told us to wait for the prim…

[deleted]

Re: Slack’s Outage on January 4th 2021

#86

Post by gen220 »

jrockway wrote: post #62

The thing that always worries me about cloud systems are the hidden dependencies in your cloud provider that work until they don't. They typically don't output logs and metrics, so you have no choice to pray that someone looks at your support ticket and clicks their internal system's "fix it for this customer" button. I'll also say that I'm interested in ubiquitous mTLS so that you don't have to isolate teams with VP…

Some customers have a hard requirement that their slack instances be behind a unique VPC. Other customers are easier to sell to if you sprinkle some “you’ll get your own closed network” on top of the offer, if security is something they’ve been burned by in the past.

I agree with you the mTLS is the future. It exists within many companies internally (as a VPC alternative!) and works great. There’s some problems around the certificate issuer being a central point of failure, but these are known problems with well-understood solutions.

I think there’s mostly a non-technical barrier to be overcome here, where the non-technical executives need to understand that closed network != better security. mTLS’s time in the sun will only come when the aforementioned sales pitch is less effective (or even counterproductive!) for Enterprise Inc., I think.

Re: Slack’s Outage on January 4th 2021

#87

Post by gen220 »

whatisthiseven wrote: post #19
saurik wrote: post #7

Earlier quoted context omitted.

I miss EC2 Classic :/. It always feels like the entire world of VPCs must have come from the armies of network engineers who felt like if the world didn't support all of the complexity they had designed to fix a problem EC2 no longer had--the tyranny of cables and hubs and devices acting as routers--that maybe they would be out of a job or something, and so rather than design hierarchical security groups Amazon just…

Agreed, I always thought VPC and all that complexity was a big step backwards. My org is moving from a largely managed network into AWS, and now we have to configure the whole network and external gateways ourselves? What engineer wants to do this? VPCs are virtual, but I don't need VPCs, I need the entire network layer virtualized and abstracted. As you suggested,just grouping devices in a single network and saying…

People use it as a (imo lazy) form of enforcing access control. If two services aren’t in the same VPC, they can’t talk to each other. It theoretically limits the damage of a rogue node.

Of course, it also creates a ton of overhead and complexity, because you still have to wire all your VPCs together to implement things like monitoring and log aggregation, for example.

As other people have suggested, the better solution (imo) is to have all your traffic be encrypted with mTLS, and enforce your ACLs with certs instead of network accessibility.

Re: Slack’s Outage on January 4th 2021

#88

Post by 0dmethz »

sokoloff wrote: post #79
0dmethz wrote: post #36

Earlier quoted context omitted.

I mean, when the hammer manufacturer sells managed, auto scaling thumb-avoiding services you might rely on that. If I understand correctly they didn't initially hit a TGW quota, it just didn't scale up fast enough.

"Hey Boss, this system that our team selected and configured, behaved as documented but not in a way that protected our customers' experience. It's Amazon's fault, not ours..." If someone came to me with that, I'd educate them on how I saw it quite differently, politely but firmly.

Unless I'm misunderstanding something the system did not perform as documented. It should have scaled, it didn't.

When a critical piece of infrastructure fails under massive load I'm not sure it it'll help much when you politely tell your engineers they fucked up for not anticipating it.

You learn lessons. Both Slack and AWS seem to have learnt lessons here.

Re: Slack’s Outage on January 4th 2021

#89

Post by JMTQp8lwXL »

tw04 wrote: post #76
JMTQp8lwXL wrote: post #59

Earlier quoted context omitted.

You have to know how to write code that fits into the cloud. You can't arbitrarily read/write to the file system, acting as if there's only one instance of the server running (if you plan to run hundreds or thousands). So even by waving the cloud 'magic wand', you still need to understand writing code in a cloud-friendly way. So in some sense, it's a shared responsibility between the vendor and engineering. You need…

Per the article, literally nothing in their code would have solved the issue. AWS was supposed to auto-scale TGWs and didn't. >Our own serving systems scale quickly to meet these kinds of peaks in demand (and have always done so successfully after the holidays in previous years). However, our TGWs did not scale fast enough. During the incident, AWS engineers were alerted to our packet drops by their own internal moni…

Correct, I was disputing the point that you can freely code without being mindful of the architecture even though the selling point of cloud providers is "focus on code, leave architecture to us". I'm not disputing in this case AWS was at fault: as the customer, Slack did everything right.

Re: Slack’s Outage on January 4th 2021

#90

Post by [deleted] »

fipar wrote: post #27
cett wrote: post #13

Earlier quoted context omitted.

Though the nuance is Slack did know how to handle it, AWS didn't.

I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. This usually works well, under the rationale that "upstream provider does this for a living, so they must be better than us at this", but if you have too unique needs (or are just a bit "unlucky"), it can fail too. All this to say that the cloud isn't magic. From a risk/e…

[deleted]
Post reply on HN