71–80 of 110 posts
I really enjoy reading these write-ups, even if the causal incident is not something I enjoy.
Not at all fun when you're actively involved in mitigating these, though. Pretty rough and sometimes scars you for life.
Hearing about others' similar experiences makes me feel a connection to them, and often teaches me something.
Earlier quoted context omitted.
Not at all fun when you're actively involved in mitigating these, though. Pretty rough and sometimes scars you for life.
I work on critical services, similar to this. While nobody likes the overall business impact of events like this, I love being in the trenches when things like this happen. I enjoy the pressure of having to use anything I can to mitigate as quickly as possible. I used to work in professional kitchens before software, and it feels a lot like the pressure of a really busy night as a line cook. Some people love it.
Earlier quoted context omitted.
Though the nuance is Slack did know how to handle it, AWS didn't.
If your oldest request was queued 5+ seconds ago in a near-realtime system (such as Slack), CPU usage isn't your biggest problem. Slack wrote an autoscaling implementation that ignored request queue depth and downsized their cluster based on CPU usage alone, so while they knew how to resolve it, I would not go so far as to say they knew how to prevent it. The mistake of ignoring the maxage of the request queue is per…
What's the first?
> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…
Though the nuance is Slack did know how to handle it, AWS didn't.
They explained to me that they'd intentionally slam the production website with external traffic a couple of times per year, at a scheduled time in the middle of the night. Like basically an order of magnitude greater than they'd every received in real life, just to try to find the breaking point. The production website would usually go down for a bit, but this was vastly better than the website going down when actual real users are trying to sign up for the Boston Marathon.
Slack probably should've anticipated this surge in traffic after the holidays, and if might have been able to run some better simulations and fire drills before it occurred.
Earlier quoted context omitted.
>I don't mean this ironically, but I think Slack did not actually know how to handle it: they outsourced the handling of this; they passed the buck. Isn't that literally supposed to be the sales pitch for the cloud? Get away from the infrastructure as a whole so you can focus on code, and let the cloud providers wave their magic wand to enable scaling? If you're saying now the story is: well rely on them to auto scal…
You have to know how to write code that fits into the cloud. You can't arbitrarily read/write to the file system, acting as if there's only one instance of the server running (if you plan to run hundreds or thousands). So even by waving the cloud 'magic wand', you still need to understand writing code in a cloud-friendly way. So in some sense, it's a shared responsibility between the vendor and engineering. You need…
>Our own serving systems scale quickly to meet these kinds of peaks in demand (and have always done so successfully after the holidays in previous years). However, our TGWs did not scale fast enough. During the incident, AWS engineers were alerted to our packet drops by their own internal monitoring, and increased our TGW capacity manually. By 10:40am PST that change had rolled out across all Availability Zones and our network returned to normal, as did our error rates and latency.
The thing that always worries me about cloud systems are the hidden dependencies in your cloud provider that work until they don't. They typically don't output logs and metrics, so you have no choice to pray that someone looks at your support ticket and clicks their internal system's "fix it for this customer" button. I'll also say that I'm interested in ubiquitous mTLS so that you don't have to isolate teams with VP…
Earlier quoted context omitted.
If you hit yourself on the thumb while using a hammer, do you blame the hammer manufacturer or yourself? TGW limits are well documented.
I mean, when the hammer manufacturer sells managed, auto scaling thumb-avoiding services you might rely on that. If I understand correctly they didn't initially hit a TGW quota, it just didn't scale up fast enough.
It's Amazon's fault, not ours..."
If someone came to me with that, I'd educate them on how I saw it quite differently, politely but firmly.
> On January 4th, one of our Transit Gateways became overloaded. The TGWs are managed by AWS and are intended to scale transparently to us. However, Slack’s annual traffic pattern is a little unusual: Traffic is lower over the holidays, as everyone disconnects from work (good job on the work-life balance, Slack users!). On the first Monday back, client caches are cold and clients pull down more data than usual on the…
Though the nuance is Slack did know how to handle it, AWS didn't.
Or maybe their monitoring and response staff was just coming back online.