Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

61–70 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#61

Post by henriquez »

sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

I’ve anecdotally experienced success doing this; the only downside is cost. In many cases you end up throwing money/hardware at a problem, but having used PaaS for high scale apps where money was no issue was like life on easy mode.

Re: A terrible, horrible, no-good, very bad day at Slack

#62

Post by ex_amazon_sde »

harpratap wrote: post #15
hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

> Chaos Engineering

...or just "availability testing" or simply "testing".

Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices.

For example: saturating the uplink bandwidth on a whole datacenter.

Re: A terrible, horrible, no-good, very bad day at Slack

#63

Post by bigiain »

sudhirj wrote: post #58
WJW wrote: post #39

Earlier quoted context omitted.

The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.

This is actually a plus point if you ask me. These few thousand customers are all operating on the same racks as me - in an environment like AWS or GCP there's no special part of the datacenter reserved for fancy customers. The billion-dollar customers all run their VMs on the same rack as me. So whatever work the ops teams do to keep things reliable benefits me as much as the biggest customers.

Slack's problem though had nothing to do with racks going down.

I'm not convinced Amazon's team is immune from the sort of complex failure mode described here. 'll bet there's people with equivalent sorts of stories about where edge cases in service interactions (either their own set of Lambda services or the AWS ones behind them, or more likely both) lead to a similar unexpected failure cascade.

Re: A terrible, horrible, no-good, very bad day at Slack

#64

Post by bigiain »

foota wrote: post #54
jwr wrote: post #36

Earlier quoted context omitted.

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

What? You can totally run disturbed SQL without data loss. Did you mean to imply something in particular?

CAP theorem strongly suggests if you optimise against data loss you have to trade off something else which is probably important too...

Re: A terrible, horrible, no-good, very bad day at Slack

#65

Post by cranekam »

harpratap wrote: post #15
hyperman1 wrote: post #4

I've just been bitten by this too: The broken monitoring hadn’t been noticed partly because this system ‘just worked’ for a long time, and didn’t require any change. Any experience on how to deal with it? Who watches the watchers?

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

But how, exactly?

Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production.

In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures.

It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.

Re: A terrible, horrible, no-good, very bad day at Slack

#66

Post by aflag »

jwr wrote: post #36
sudhirj wrote: post #31

This is one of the biggest arguments I see for serverless (AWS Lambda + DynamoDB) or at least managed PaaS systems (Google App Engine, Heroku with RDS or CloudSQL). These systems may seem to cost more for some workload curves (or might even be cheaper for your curve), but the difference is worth it because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and b…

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

Though they do have good availability, they are certainly not infallible and have been down for hours or even days. And, when that happens, all you can do is pray they get to it soon. You have no control over when they will make riskier changes or how fast they will be able to respond. If they fail to respect the SLA, the maximum they are going to do is giving your money back for your services.

Obviously, there are good aspects of outsourcing devops/admin work. It's a tradeoff, as most things. If you are a struggling startup it's difficult to justify the cost of owning a bunch of hardware and hiring expensive infra people to manage it. However, facebook is probably better off owning their own infraestructure.

Re: A terrible, horrible, no-good, very bad day at Slack

#67

Post by blahbhthrow3748 »

sudhirj wrote: post #58
WJW wrote: post #39

Earlier quoted context omitted.

The specialized 24/7 devops team (if it's there) also has a few thousand other customers instead of being there just for you. They might have other priorities at any given moment. It's not like AWS or GCP are renowned for the quality of their customer service.

This is actually a plus point if you ask me. These few thousand customers are all operating on the same racks as me - in an environment like AWS or GCP there's no special part of the datacenter reserved for fancy customers. The billion-dollar customers all run their VMs on the same rack as me. So whatever work the ops teams do to keep things reliable benefits me as much as the biggest customers.

In my experience the ops team will be constantly changing things and upgrading to keep all those customers happy, and they don't really care about your personal risk tolerance. They probably screw up a smaller proportion of the changes but they make an order of magnitude more and you can't predict what impact it will eventually have

Re: A terrible, horrible, no-good, very bad day at Slack

#68

Post by bigiain »

user5994461 wrote: post #49
nik_0_0 wrote: post #7

Super interesting post. Following blog links, the timeline in https://slack.engineering/all-hands-on-deck-91d6986c3ee also offers a look at the play by play. However, as far as I can read it, they have somewhat different views on the root cause? "Soon, it became clear we had stale HAProxy configuration files, as a result of linting errors preventing re-rendering of the configuration." vs. "The program which synced th…

The way they are doing things. HAProxy is configured with a fixed amount of slots. This effectively acts as a maximum limit, so should be enough for the running instances + newer instances coming up anytime due to auto scaling. They have a tool listening to applications starting and shutting down. It's adjusting the configuration live while running to remove shut down instances (free a slot) and put in newer instance…

"640K ought to be enough for anybody!"

Eventually, seemingly sane assumptions become anachronistic laughing points. (Even if they're apocryphal...)

Re: A terrible, horrible, no-good, very bad day at Slack

#69

Post by rutthenut »

foota wrote: post #54
jwr wrote: post #36

Earlier quoted context omitted.

> because you're paying for specialized 24/7 dev-ops teams whose only job is to keep these systems running smoothly, and by definition they're already familiar with running workloads orders of magnitude bigger than yours This is based on faith — there might, or might not be a specialized 24/7 devops team who runs these things better than you. My rational mind has trouble accepting things based on faith, which is also…

What? You can totally run disturbed SQL without data loss. Did you mean to imply something in particular?

'Disturbed' SQL confused me, had to look up the thread to see Distributed SQL. And yes, I agree, you certainly can run distributed without data loss, depending on the platform design and your 'transaction level'.

Re: A terrible, horrible, no-good, very bad day at Slack

#70

Post by antoinealb »

tinco wrote: post #40
jacquesm wrote: post #12

Earlier quoted context omitted.

This same sort of thing tends to creep into backup systems that work for a long time without if the backups are never restored. The more reliable the systems they backup the bigger the chance that a restore will fail when you need it most. So test your backups .

Would you consider it evil to use production backups for the QA step in CI/CD? It would catch two birds with one stone, continuously verifying the backups, and ensuring the new code works on real world data. We don't have any personal information in our production database, but even if we did, as long as the QA is thoroughly prevented from interacting with the outside world, it can't hurt to use production data right…

I think it is a big issue to use anything containing PII in anything else than production environment. For example, do your QA logs have the same kind of management, access control and lifetime than production ? If they don't, you might end up logging PII in a way you should not, and could even be in legal troubles for things like GDPR.
Post reply on HN