Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack is down

status.slack.com

Post reply on HN
Print view View original

631–640 of 840 posts

Re: Slack is down

#631

Post by thesuitonym »

MattGaiser wrote: post #351
blntechie wrote: post #332

I have always had this fantasy thinking of what happens when outages of one of these major service never come back online i.e. in this outage Slack loses info of all the accounts, users, messages etc. How would people react? What would engineers do to recover? I always found that idea fascinating. Imagine Google saying tomorrow that they lost all the accounts and emails. What kind of impact the world will have?

Google would be catastrophic because so much is stored there. Slack is mostly real time communication, at least for me. There are a few bits and bobs that really should be documented that are in the messages though.

If this thread is to be believed, apparently a lot of engineers use slack for alerting and don't know how to check their monitoring software manually.

Re: Slack is down

#632

Post by cgh »

sytse wrote: post #258

At GitLab our fallback from Slack is Zoom https://about.gitlab.com/handbook/communication/#emergency-c... I'm posting this because I found a lot of people don't know that Zoom includes a complete chat client that includes channels. And #HugOps to the engineers at Slack working on this. I appreciate that they posted a periodic update even when there was no news to report: "There are no changes to report as of yet. We'…

Unpopular opinion, but WebEx beats the pants off Zoom. Of course, it's neither free nor open. But it does support strong end to end encryption and authentication and has regulatory compliance to a bunch of things, if that's important to you. I get that there is WebEx hate because "enterprise" etc, but we use it around here and it works quite well.

Re: Slack is down

#633

Post by Hnrobert42 »

trastknast wrote: post #629
daniellarusso wrote: post #467

Earlier quoted context omitted.

Maybe my DevOps folks should not be privy to all internal communications? That is one reason we did not go with Mattermost.

Slack won’t protect you from this as it’s possible for admins to export even private DMs. https://www.nbcnews.com/better/business/slack-updates-privac...

Only Slack Workspace Owners can export, not Slack Admins.

Re: Slack is down

#634

Post by thesuitonym »

I'd like to take this moment to mention self-hosted, open source, and federated alternatives like XMPP and Matrix.

I'd like to, but unfortunately I don't feel like I can in good faith. Matrix is woefully immature, and suffers from a lot of issues, but I think is closer to being a functional Slack/Discord alternative. XMPP is much more mature, and works very well for chat, but doesn't have a nice package that does all the Slack stuff--at least not that I'm aware of. I'd love to be proven wrong there. I know it can be done, but if it can't be deployed quickly by an already overstressed team member, what chance does it have?

Re: Slack is down

#635

Post by aledalgrande »

I think this is due to AWS. Not only Slack is down (e.g. Notion). AWS status page didn't show anything yet, but wouldn't be the first time. The last Kinesis crisis didn't show up for hours.

Re: Slack is down

#636

Post by bobthepanda »

nkassis wrote: post #105
exhaze wrote: post #89

When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident. My bet is that this incident is caused by a big release after a post-holiday "code freeze".

I would bet it's just the influx of traffic post holiday with systems that haven't been updated in so long maybe some annoying memory leaks have crept up and gone unnoticed or some other bad state that was exacerbated by return to work day for most NA folks. Code freezes were good at identifying bugs that only show up after long periods. Doubt anyone releasing big changes Monday morning.

What would make that strange? Where I work it is frowned upon to do releases on weekends and so bad changes due to buildups happen on Monday.

Although, we also don’t close the pipeline for just any holiday break. In fact low holiday traffic is a good time to keep pipelines open, since changes will impact less people.

Re: Slack is down

#637

Post by lima »

cgh wrote: post #632
sytse wrote: post #258

At GitLab our fallback from Slack is Zoom https://about.gitlab.com/handbook/communication/#emergency-c... I'm posting this because I found a lot of people don't know that Zoom includes a complete chat client that includes channels. And #HugOps to the engineers at Slack working on this. I appreciate that they posted a periodic update even when there was no news to report: "There are no changes to report as of yet. We'…

Unpopular opinion, but WebEx beats the pants off Zoom. Of course, it's neither free nor open. But it does support strong end to end encryption and authentication and has regulatory compliance to a bunch of things, if that's important to you. I get that there is WebEx hate because "enterprise" etc, but we use it around here and it works quite well.

+1, both are a PITA but Webex at least has a really good web client.

Re: Slack is down

#638

Post by helper »

jrockway wrote: post #44
schoolornot wrote: post #8

How many more outages until all trust is eroded and competing services differentiate themselves on the basis of uptime?

I say this every time Slack is down, but they just seem so shady to me. Nobody can connect right now, and their status site says "100% uptime in the last quarter". Maybe it's close to 100%, but it ain't 100%. I think we should push for a metric where "up" means 100% of people that want to use the service are able to use the service. If 1% of users can't send messages, then that should count as a full-blown outage and…

That number is almost certainly updated manually. Check back tomorrow and see what it says.

If you look at the history page you can see its not 100% for every month: https://status.slack.com/calendar

Re: Slack is down

#639

Post by jcranmer »

blntechie wrote: post #461
jacques_chester wrote: post #443

Earlier quoted context omitted.

Yes, Google plans extensively and runs regular drills. It's hearsay, but I was once told that achieving "black start" capability was a program that took many years and about a billion dollars. But they (probably) have it now.

So 'black start' is a program to start over from scratch? The scale required for it itself would be amazing.

"Black start" is a term that refers to bringing up services when literally everything is down.

It's most often referred to in the electricity sector, where bringing power up after a major regional blackout (think 2003 NE blackout) is extremely nontrivial, since the normal steps to turn on a power plant usually requires power: for example, operating valves in a hydro plant or blowers in a coal/gas/oil plant, synchronizing your generation with grid frequency, having something to consume the power; even operating the relays and circuit breakers to connect to the grid may require grid power.

The idea here is presumably that Google services have so many mutual dependencies that if everything were to go down, restarting would be nontrivial because every service would be blocked on starting up due to some other service not being available.

Re: Slack is down

#640

Post by pachico »

thesuitonym wrote: post #634

I'd like to take this moment to mention self-hosted, open source, and federated alternatives like XMPP and Matrix. I'd like to, but unfortunately I don't feel like I can in good faith. Matrix is woefully immature, and suffers from a lot of issues, but I think is closer to being a functional Slack/Discord alternative. XMPP is much more mature, and works very well for chat, but doesn't have a nice package that does all…

I can hardly see those as alternatives to Slack. Maybe https://mattermost.com/ is what you were thinking about?
Post reply on HN