Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

71–80 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#71

Earlier quoted context omitted.

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

But how, exactly? Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could d…

> Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question

No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.

Re: A terrible, horrible, no-good, very bad day at Slack

#72
post #66
post #36

Earlier quoted context omitted.

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

Though they do have good availability, they are certainly not infallible and have been down for hours or even days. And, when that happens, all you can do is pray they get to it soon. You have no control over when they will make riskier changes or how fast they will be able to respond. If they fail to respect the SLA, the maximum they are going to do is giving your money back for your services. Obviously, there are g…

FWIW, that time AWS had a massive massive failure it was actually really convenient being on AWS as so much of the Internet was down at the same time that pretty much no one was upset at me for my one service also being offline.

Re: A terrible, horrible, no-good, very bad day at Slack

#73

Earlier quoted context omitted.

AWS is pretty good whenever we've needed them. Google? Probably not.

Google swears gcloud support is good, of course, but I've never actually used it -- but then again, I've never actually needed it. Meanwhile, I need AWS support constantly because their entire platform is a gigantic social experiment in minimum viable products. How crusty are people willing to tolerate? Evidently: very, very crusty.

Vice versa for us. Google always has troubles and outages and AWS worked fine. Not sure why.

Re: A terrible, horrible, no-good, very bad day at Slack

#74

Earlier quoted context omitted.

AWS is pretty good whenever we've needed them. Google? Probably not.

Google swears gcloud support is good, of course, but I've never actually used it -- but then again, I've never actually needed it. Meanwhile, I need AWS support constantly because their entire platform is a gigantic social experiment in minimum viable products. How crusty are people willing to tolerate? Evidently: very, very crusty.

We don't use most of the platform but the stuff we do use is rock solid (VPC, EC2, RDS, Route53, S3 etc). I do sometimes wonder what the point is for half the stuff they release.

What I do know about GCP is they had a production bug in their tooling that was breaking everything for literally 100s of customers and they never even bothered replying to the bug report on their support forums. That experience along with the general modus operandi of shutting things down that don't further their surveillance capitalism business model means I won't be trying them again.

Re: A terrible, horrible, no-good, very bad day at Slack

#76
post #75

>terrible, horrible, no-good, bad day Traditionally, a "black day". I guess that's not OK anymore, but I find the effort to work around established language cute. Edit: Apparently a meme.

https://en.wikipedia.org/wiki/Alexander_and_the_Terrible,_Ho...

Re: A terrible, horrible, no-good, very bad day at Slack

#77
post #75

>terrible, horrible, no-good, bad day Traditionally, a "black day". I guess that's not OK anymore, but I find the effort to work around established language cute. Edit: Apparently a meme.

It's a reference to Alexander and the Terrible, Horrible, No Good, Very Bad Day.

I find the effort to make something innocuous into a confirmation of personal prejudices cute.

Re: A terrible, horrible, no-good, very bad day at Slack

#80
post #63
post #58

Earlier quoted context omitted.

This is actually a plus point if you ask me. These few thousand customers are all operating on the same racks as me - in an environment like AWS or GCP there's no special part of the datacenter reserved for fancy customers. The billion-dollar customers all run their VMs on the same rack as me. So whatever work the ops teams do to keep things reliable benefits me as much as the biggest customers.

Slack's problem though had nothing to do with racks going down. I'm not convinced Amazon's team is immune from the sort of complex failure mode described here. 'll bet there's people with equivalent sorts of stories about where edge cases in service interactions (either their own set of Lambda services or the AWS ones behind them, or more likely both) lead to a similar unexpected failure cascade.

> I'm not convinced Amazon's team is immune from the sort of complex failure mode

You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive. Like, hourly cron job doubling / halving within narrow safety rails primitive, where it's not merely possible to find a load that trips it up, it's all but inevitable.

This varies by service, but they always project an image of their infrastructure being rather smart, and in the cases where I've been able to make an informed guess about what's actually going on, it's usually wildly inconsistent with the marketing. They don't warn you about the stinkers and even on services with good autoscaling and no true incompatibility between AWS's hidden choices and your needs, your scaling journey will involve periodic downtime as you trip over hidden built in limits and have to beg support to raise them. Sometimes you get curiously high resistance to this, leading to the impression that these aren't so much "safety rails" as hardcoded choices.

Oh, and just last week we managed to completely wedge a service. The combination of a low limit on in-flight processes, two hung processes, immutability on running processes, and delete functionality being predicated on proper termination led to a situation where an AWS service became completely unusable for days while we begged support to log in and clear the hung process. Naturally, this isn't going to count as downtime on any reliability charts, even though it's a known problem and definitely looked a lot like downtime on our end.

We're a small (<10) team with modest needs. AWS lets us do some crazy awesome things, but it really bugs me how reliably they over-promise and under-deliver.

Post reply on HN