Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A terrible, horrible, no-good, very bad day at Slack

slack.engineering

Post reply on HN
Print view View original

241–250 of 282 posts

Re: A terrible, horrible, no-good, very bad day at Slack

#241

Post by sailfast »

debt wrote: post #217
m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

TL;DR meta point is: "The reason that we haven’t been doing any significant work on this HAProxy stack is that we’re moving towards Envoy Proxy for all of our ingress load-balancing" The legacy system supporting Slack in production was heavily resource-constrained as they were moving to a new fancy system. Slack admits here that the legacy system likely wasn't getting the attention it needed and lo-and-behold it star…

I really feel this comment. Especially during transition it’s hard to see the “to-be legacy” system as worthy of the effort because the new stuff will be here so soon! Until it isn’t because life happens and you’re left with a really shaky platform. These are tough investment decisions especially when resource constrained. As a rule, perhaps ensuring that current state platforms are secure, as you say, before attempting a migration is the best way to go, but one person’s critical work is another person’s “too much insurance.”

Re: A terrible, horrible, no-good, very bad day at Slack

#242

Post by jzoch »

vondur wrote: post #194
m12k wrote: post #26

TL;DR First a performance bug was caught during rollout, and rolled back within a few minutes. However this triggered their auto-scaling of web apps to ramp up to more instances than a hard limit they had. This in turn triggered a bug in how they update the list of hosts in their load balancer, causing it to not get updated with new instances, and eventually go stale. After 8 hours the only real remaining instances i…

It almost seems like no one group understands the system as a whole, so when one part fails, no one has a clear idea of the domino effects that can happen. I'm guessing this is the result of really complex systems interactions.

Its a load balancer bug. Thats it. Everything else functioned totally normally from what I can see. I think people are blowing this out of proportion saying everything is so complex or its all microservices oh no!

This exact bug could have bit a large monolithic app running on a VM.

Re: A terrible, horrible, no-good, very bad day at Slack

#243

Post by dannyw »

throwaway0a5e wrote: post #37
rkagerer wrote: post #25

Earlier quoted context omitted.

There's a reason the military does drills. Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. It's especially critical for disaster recovery (a DR plan that's never tested isn't worth the paper it's written on). Consider going one step further and randomly injecting artificial failures into production shards, so handling them becomes a regular affair…

>Do them regularly, and keep your playbook of mock failure scenarios up to date with good coverage of all your systems. All those man hours cost money. Is it worth it? Depends on how much downtime costs you.

If you run a fairly discretionary web app, like perhaps a game where downtime will cost you lost revenue for the time period but that's it, it's probably not worth it.

If you are running a communications platform like Slack or Gmail, it's hella worth it.

Re: A terrible, horrible, no-good, very bad day at Slack

#244

Post by dannyw »

anderspitman wrote: post #109

Seems to me that most of the problems came from the sheer scale Slack operates at. A single instance of a self-hosted chat application wouldn't require any of the load balancing infrastructure. But SaaS is really convenient. I wonder if there's a market for "local cloud" companies that operate out of your city and offer hosting of popular open source projects. The complexity would be much lower, and hopefully the rel…

A single instance of a self-hosted chat application would also result in unbounded hours investigating some obscure bug that affects 1 in 100,000 deployments, etc; while the developers don't know because they don't have enough ability to remotely dig in and investigate.

Self hosting is not a pancea and I would not think it would be more reliable.

There's also limited reason why you really want your servers in your city, 5ms of latency savings isn't it,instead of economies of scale in large datacentres with good network uplinks and centralized reliability teams.

Re: A terrible, horrible, no-good, very bad day at Slack

#245

Post by jjoonathan »

bigiain wrote: post #239
jjoonathan wrote: post #80

Earlier quoted context omitted.

> I'm not convinced Amazon's team is immune from the sort of complex failure mode You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive. Like, hourly cron job doubling / halving within narrow safety rails primitive, where it's not merely possible to find a load that trips it up, it's all but inevitable. This varies by service, but they always project an image of their i…

> You're being way too kind. Not only is AWS not immune, their autoscalers are often absurdly primitive. Yep. Very much so. Mostly because I don't have enough personal Lambda-specific warstories to feel confident badmouthing it in the context of this discussion thread. But the bits of AWS I do use are certainly not all rainbows and roses... I have one app/platform I run that basically sits at a few requests an hour f…

Hey, let's not be hasty, it sounds like you've built a bespoke autoscaling backend that intelligently predicts future usage and dynamically allocates compute resources to match customer needs.

Re: A terrible, horrible, no-good, very bad day at Slack

#246

Post by matiasfernandez »

fireflux_ wrote: post #226
matiasfernandez wrote: post #221

"I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?" Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID…

Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ I revisit this video every now and then.

It's all I could think of while reading the article. The spiraling-down into complexity and service names had me smiling all the way. The "we should have predicted this" in last paragraph sealed the deal. And the HN comments saying they should have known better and used ZingBing from the start had me rolling.

"You think you know what it takes to tell the user it's their birthday?!"

Re: A terrible, horrible, no-good, very bad day at Slack

#247

Post by peterwwillis »

cutemonster wrote: post #187
peterwwillis wrote: post #131

Earlier quoted context omitted.

Google goes over this scenario in their SRE books. You literally plan to take down your service so you never have 100% uptime, and literally step through a real failure and recovery. In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask t…

What is, according to you, and the SRE book, a good time during the day/week/year to do these real test outages? How much downtime could be ok? For, let's say, a b2b saas They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)

It doesn't have to be a customer impact to bring down a service or parts of a service (if it's designed correctly). But even if it was a customer impact, you can do it at the same time you'd schedule regular maintenance (there's always something that needs maintenance, eventually) and throw up a maintenance page.

It's up to the business to define SLAs and SLOs that the customer will be satisfied with.

Re: A terrible, horrible, no-good, very bad day at Slack

#248

Post by celim307 »

fireflux_ wrote: post #226
matiasfernandez wrote: post #221

"I'm still not understanding why it's so hard to display the birthday date on the settings page. Why can't we get this done this quarter?" Look, I'm sorry, we've been over this. It's the design of our back-end. First there's this thing called the Bingo service. See, Bingo knows everyone's name-o, so we get the user's ID out of there. And from Bingo, we can call Papaya and MBS (Magic Baby Service) to get that user ID…

Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ I revisit this video every now and then.

This one always cuts too close to home

https://youtu.be/_o7qjN3KF8U

Re: A terrible, horrible, no-good, very bad day at Slack

#249

Post by matiasfernandez »

celim307 wrote: post #248
fireflux_ wrote: post #226

Earlier quoted context omitted.

Reference: https://www.youtube.com/watch?v=y8OnoxKotPQ I revisit this video every now and then.

This one always cuts too close to home https://youtu.be/_o7qjN3KF8U

Can't wait for BallmerCon this year. The XLOOKUP meta is going to spice things up.

https://www.youtube.com/watch?v=xubbVvKbUfY

Re: A terrible, horrible, no-good, very bad day at Slack

#250

Post by ripvanwinkle »

ChrisMarshallNY wrote: post #35
thdrdt wrote: post #32

Earlier quoted context omitted.

Yep. I once worked at a company that had backups of all the data. Until they needed a backup and discovered they were not readable. Months of work were gone.

This has happened to me. Long story. Incompetent (and dishonest) IT person. In fact, I was attacked for “harassing” the IT person. Nothing was done about it, until the HR DB got borked, and there was no backup. DR and backup are profoundly unpopular topics. They tend to be expensive, and difficult to test. They also presuppose a Very Bad Thing happening, which no one wants to think about. I have scars. I now have mul…

there's a joke in the data protection industry.

Backups always work, it's the restores that have problems

Post reply on HN