Slack is down

Post by radicalbyte » Mon, Jan 04, 2021, 4:17 PM UTC

hnlmorg wrote: post #113

nkassis wrote: post #105

Earlier quoted context omitted.

I would bet it's just the influx of traffic post holiday with systems that haven't been updated in so long maybe some annoying memory leaks have crept up and gone unnoticed or some other bad state that was exacerbated by return to work day for most NA folks. Code freezes were good at identifying bugs that only show up after long periods. Doubt anyone releasing big changes Monday morning.

That might be true but when you take the global usage of Slack and their respective time zones, more than half the world would have signed into Slack this morning before SV had and I certainly didn't notice any downtime this morning in my time zone.

Post by ghostpepper » Mon, Jan 04, 2021, 4:17 PM UTC

JMTQp8lwXL wrote: post #42

lol768 wrote: post #17

These events seem to be happening almost on a monthly basis now. IRC was never this unreliable and at least with netsplits it was obvious what had happened because you'd see the clients disconnect. IME messages just fail to send with Slack, then you can retry but they're not properly idempotent and you end up sending the messages twice. It's really poor.

To be fair, IRC doesn't do a lot of things Slack does. Where is the logging and audit trails, access control, search, etc.

Post by kevinmchugh » Mon, Jan 04, 2021, 4:18 PM UTC

exhaze wrote: post #126

exhaze wrote: post #89

When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".

To elaborate a bit more on this point, you have to think about it like any complex system failure - it's almost never one thing, but rather a combination of many different factors. The factors around post NYE releases: - high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, t…

Post by iso1631 » Mon, Jan 04, 2021, 4:19 PM UTC

TeMPOraL wrote: post #30

Snitch-Thursday wrote: post #2

Using our teams backup chatroom in a competing service. One of these days P2P Matrix will reach GA, then I plan to make a backup for my backups, Starfleet style.

That's one obscure reference, I love it. https://www.youtube.com/watch?v=UaPkSU8DNfY GILORA: Starfleet code requires a second backup? O'BRIEN: In case the first backup fails. GILORA: What are the chances that both a primary system and its backup would fail at the same time? O'BRIEN: It's very unlikely, but in a crunch I wouldn't like to be caught without a second backup.

  LOJAL: I've been reading the reports of your Chief of Operations, Doctor. They gave me the impression that he was a competent engineer.
  BASHIR: Chief O'Brien? One of the best in Starfleet.
  LOJAL: Then why aren't the backup systems functioning?
  BASHIR: Well, you know, out here on the edge of the frontier, it's one adventure after another. Why don't I escort you back to your quarters where I'm sure we can all wait this out. 

  KIRA: My terminal just self-destructed.
  DAX: What?
  KIRA: I lost an evaluation report I've been working on for weeks.
  DAX: Even the backups?
  KIRA: Even the backups. 

Post by sbilstein » Mon, Jan 04, 2021, 4:19 PM UTC

zwily wrote: post #144

rwc wrote: post #138

Earlier quoted context omitted.

Seems to be more than that. Even slack.com in an incognito browser fails.

What does an incognito browser have to do with anything?