371–380 of 840 posts
At GitLab our fallback from Slack is Zoom https://about.gitlab.com/handbook/communication/#emergency-c... I'm posting this because I found a lot of people don't know that Zoom includes a complete chat client that includes channels. And #HugOps to the engineers at Slack working on this. I appreciate that they posted a periodic update even when there was no news to report: "There are no changes to report as of yet. We'…
When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".
(I'm not making a statement if that's good or bad or if it works or whatever. Please don't read an opinion into it.)
For some reason, today, HN seems exceedingly exceedingly slow (to me) after logging in... Without being logged in, things are as fast as they usually are -- but post log-in, SLOWWWER THAN MOLASSESS ... I tried this several times; why this is, I can only wonder... To quote Bill and Ted... "Strange things are afoot at the Circle-K..."
Having been in this situation before, with a totally-down-and-not-coming-back-up outage of a payments system, I really feel for their incident response team. I'll take this moment to remind everyone of their human tendency to read meaning into random events. There's no evidence to suggest New Year traffic has caused this, and outages like this can happen in spite of professional and competent preparation. Hugops for…
On the one hand, sure we don't specifically know what's going on. On the other hand, it's the first Monday in the new year and they went down shortly after the start of the business day Eastern time; it could be coincidence, but it would be a remarkable coincidence.
> Customers may have trouble connecting or using Slack I can't stand how marketing speak pervades every sphere of the world. Their entire system is offline (inconvenient certainly, but it happens) and they can't bring themselves to say "Slack is down. We're working on it and will be back ASAP." or something similar. Instead we may have trouble.
When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".
To elaborate a bit more on this point, you have to think about it like any complex system failure - it's almost never one thing, but rather a combination of many different factors. The factors around post NYE releases: - high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, t…
People came back to work, and most of them start around the same time (US wise at least).
Hence kids - a vital lesson for all of us - don't start the call at a full hour, give it 3-7 min to make your coworkers confused and give some time for the systems to auto-scale ;)
Earlier quoted context omitted.
How do you know it's down completely? Maybe it's down for you and maybe even down for a majority but still up for some subset. Happens with many products.
https://status.slack.com/ Every service is marked as "Outage" as of now (also when I wrote the comment).
> Customers may have trouble connecting or using Slack I can't stand how marketing speak pervades every sphere of the world. Their entire system is offline (inconvenient certainly, but it happens) and they can't bring themselves to say "Slack is down. We're working on it and will be back ASAP." or something similar. Instead we may have trouble.