Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Slack issues

status.slack.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Slack issues

#51

Post by ryanbrunner »

capableweb wrote: post #36
Hamuko wrote: post #35

Earlier quoted context omitted.

There is also an incentive to display the proper status, since if you show all green when the service is down, you look dumb. We recently improved our status reporting because our old status system might not accurately indicate if only a part of our service went down, so it might show green even though (some) customers definitely saw very obvious downtime.

When the choice is between temporarily "looking dumb" and permanently loosing a bunch of money because of SLAs, companies like Salesforce usually make the latter choice.

Usually this is less being outright deceptive (saying you're up when you know you're down), and more wrapping the definition of "down" in legalese so that nothing short of a complete system outage of every aspect of your service for 100% of users counts as "down".

In those cases status pages are more likely to have a lovely spectrum of yellows and oranges to choose from, but a distinct absence of red, but very few companies will say that they're fully green when they know there's issues. Green when you're down is usually more of a symptom of process problems in how outages get reported or resolved (no documentation, manual processes for updating, long communication chains from support teams to engineers).

Re: Slack issues

#52

Post by bombcar »

thecopy wrote: post #40
Hamuko wrote: post #37

Earlier quoted context omitted.

I feel like customers treating you like a bunch of fucking idiots is not gonna be a great boost to your bottom line either. Maybe if you're in a market sector where reputation doesn't matter and you don't have any competition swooping in.

The people signing the contract with Slack is not looking at the status pages, generally.

The people signing contracts ONLY look at the status page, and say everything’s fine. They don’t bother verifying it signing in.

Re: Slack issues

#53

Post by ryanbrunner »

darkerside wrote: post #39

Would a decent status page be an actually intelligent use of blockchain? We could have a global network of computers responsible for determining a consensus on whether a service is truly and actually down or not. It could be captured in an independent ledger, and ideally used as a canonical determination for status pages, SLA disputes, etc.

downdetector already does basically this effectively enough and is dramatically simpler in terms of technology. You don't really need to be overly complex about consensus for things like this - if a sufficiently large population reports something down, it's down, because the stakes aren't high enough for enough bad actors to be an issue.

Re: Slack issues

#54

Post by ryanbrunner »

stevefan1999 wrote: post #46
darkerside wrote: post #39

Would a decent status page be an actually intelligent use of blockchain? We could have a global network of computers responsible for determining a consensus on whether a service is truly and actually down or not. It could be captured in an independent ledger, and ideally used as a canonical determination for status pages, SLA disputes, etc.

what if the blockchain itself was in an inconsistent state due to 51% attack (also known as split brain consensus)

This is compounded by the fact that there's no objective truth about whether something is down (as evidenced by this thread, outages are often not global or not across the entire service)

Re: Slack issues

#55

Post by nurettin »

01acheru wrote: post #9

Another status page that sucks. Slack goes down, people start texting me about it, status page is green, HN informs me that Slack is actually down. Next time I build a status page it will simply be a static HTML with green indicators and some random metrics, looks like it is the industry standard. :facepalm:

Status page pull request is pending merge, because seniors are either fired or or overworked.

Re: Slack issues

#56

Post by sjsdaiuasgdia »

01acheru wrote: post #9

Another status page that sucks. Slack goes down, people start texting me about it, status page is green, HN informs me that Slack is actually down. Next time I build a status page it will simply be a static HTML with green indicators and some random metrics, looks like it is the industry standard. :facepalm:

Progression of status pages, from experience at a large cloud provider...

Stage 1: Status is manually set. There may be various metrics around what requires an update, and there may be one or more layers of approval needed.

Problems: Delayed or missed updates. Customers complain that you're not being honest about outages.

Stage 2: Status is automatically set based on the outcome of some monitoring check or functional test.

Problems: Any issue with the system that performs the "up or not?" source of truth test can result in a status change regardless of whether an actual problem exists. "Override automatic status updates" becomes one of the first steps performed during incident response, turning this into "status is manually set, but with extra steps". Customers complain that you're not being honest about outages and latency still sucks.

Stage 3: Status is automatically set based on a consensus of results from tests run from multiple points scattered across the public internet.

Problems: You now have a network of remote nodes to maintain yourself or pay someone else to maintain. The more reliable you want this monitoring to be, the more you need to spend. The cost justification discussions in an enterprise get harder as that cost rises. Meanwhile, many customers continue to say you're not being honest because they can't tell the difference between a local issue and an actual outage. Some customers might notice better alignment between the status page and their experience, but they're content, so they have little motivation to reach out and thank you for the honesty.

Eventually, the monitoring service gets axed because we can just manually update the status page after all.

Stage 4: Status is manually set. There may be various metrics around what requires an update, and there may be one or more layers of approval needed.

Not saying this is a great outcome, but it is an outcome that is understandable given the parameters of the situation.

Re: Slack issues

#57

Post by stevewodil »

efficax wrote: post #49
01acheru wrote: post #9

Another status page that sucks. Slack goes down, people start texting me about it, status page is green, HN informs me that Slack is actually down. Next time I build a status page it will simply be a static HTML with green indicators and some random metrics, looks like it is the industry standard. :facepalm:

status pages are rarely automated and it looks like slack was down at like 1am pacific. somebody got woken up by a page and groggily escalated and they sat there fighting the outage for 30 minutes before someone said “what about the status page”. or at least that’s how it worked at my last company

This is very accurate. Or the customer communication lead was shadowing and this was their first incident.

It’s all the same

Re: Slack issues

#58

Post by Volundr »

rwaksmunski wrote: post #50
inferiorhuman wrote: post #44

Earlier quoted context omitted.

Nah the issue isn't looking dumb, the issue is looking like a liar. What good is an SLA if your reputation is that you're going to lie to avoid paying out?

This is Salesforce we are talking about. They topped Stack Overflow as the most dreaded technology so they bribed SO not to include them anymore. They don't care if they look like liars to technical people. Technical people are not the decisions makers in their target customer base.

> They topped Stack Overflow as the most dreaded technology so they bribed SO not to include them anymore.

Wait what? And SO agreed? Where can I learn about this?

Re: Slack issues

#59

Post by ilrwbwrkhv »

slack is bitchware. a new term i use these days for software which is forced on employees because they have no other option (hence bitches). other examples of bitchware are microsoft teams, jira etc.
Post reply on HN