Live data from Hacker News

IBM Cloud was down, as well as their status page

cloud.ibm.com

71–80 of 201 posts

Re: IBM Cloud was down, as well as their status page

#71
post #17

Also still no communication from IBM that anything is wrong.

Account managers are texting, but they have no VPN access right now.

It seems all their external network connections are down. I assume people will have to drive to the data centers to fix. I really want to see a post mortem on this outage.

Re: IBM Cloud was down, as well as their status page

#72
post #52

I remember I was at an IBM sponsored hackathon around 2015 where it was a requirement to use Bluemix. Over the course of the weekend the service went down for hours 3 times. Literally this morning I was wondering what ever happened to it, like did it die a quiet death? Oh it rebranded to IBM cloud in 2017. Now this news. I think there's an eponymous law named for this sort of thing.

[deleted]

Re: IBM Cloud was down, as well as their status page

#73
post #52

I remember I was at an IBM sponsored hackathon around 2015 where it was a requirement to use Bluemix. Over the course of the weekend the service went down for hours 3 times. Literally this morning I was wondering what ever happened to it, like did it die a quiet death? Oh it rebranded to IBM cloud in 2017. Now this news. I think there's an eponymous law named for this sort of thing.

That's funny. I've had the exact same experience with Bluemix at a Hackathon in the past. It was down for almost the entire weekend, screwing all the teams that didn't pivot early enough.

Re: IBM Cloud was down, as well as their status page

#74
post #14

wonder if we'll ever get a post-mortem about this... Seems to be global

Maybe. About 3/4 of all outages get a post mortem. There's 1/4 of the time they refuse to tell us anything.

Do you actually have data on that or are you conjecturing? Because I would really love to see data about that if it exists somewhere.

Re: IBM Cloud was down, as well as their status page

#75
post #14

wonder if we'll ever get a post-mortem about this... Seems to be global

Maybe. About 3/4 of all outages get a post mortem. There's 1/4 of the time they refuse to tell us anything.

There will have to be a post mortem on this. The convention is to be as transparent as possible as to what went wrong. This helps to let current customers know that you found the problem, and have put plans in place to make sure it doesn't happen again.

The purpose of the signalling here is two fold.

1) If convincing enough (with details), you can keep current customers from moving to a competitor.

2) It also lets new customers see how you actually handle a crisis. If they can manage the crisis well enough, then you can point to this instance to prove your technical knowhow to handle their needs.

If they don't tell anything, or aren't transparent, then they can expect a mass exodus of customers.

Re: IBM Cloud was down, as well as their status page

#76

So what are HNers using IBM Cloud for and where do you see that it has an edge over AWS offerings (where an overlap exists, obviously)? (I figure either you’re in devops and you are putting out fires too busy to read this thread or you’re not and your work is halted because of the incident so you might have time to read and reply ;)

Rarely it is a just technical decision, usually money is the reason.

In small and mid size organizations the CSP gave better pricing, or they help with your sales etc

In large organizations - IBM/Oracle bundle their existing products currently being paid for any way, or account managers have great relationships with decision makers , the company already has signed up big multi year deals.

This is not just IBM, it applies to GCP/Azure/AWS as well.

Re: IBM Cloud was down, as well as their status page

#77
post #60

The most infuriating thing about this is the ZERO communication coming out of IBM Cloud. No emails. No updates to twitter. Status page down. Support lines clogged. At least give me something I can point my customers at to show them this is not due to my incompetence.

Yep, super annoying I had to link my customers to a techcrunch page :(

Re: IBM Cloud was down, as well as their status page

#78

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

Never humiliate a coworker in public. Instead say "both options were considered but ultimately it was decided to select option B for reason Y."

My professional experience tells me that the next question will be who decided for B given Y, then you answer it and then you have a target on your SRE back, I'm afraid. Remember that the trickle-down economics works only when the shit hits the fans and what trickles down is not money.

Re: IBM Cloud was down, as well as their status page

#79
post #70
post #56

Earlier quoted context omitted.

I would mirror the attitude of the person who said no originally. If they are receptive to feedback and clearly want to do better, I would be kind and explain why I had suggested it not be there in the first place and cite this as an example. If they were being adamant or denying it was their fault, I'd probably be really quiet and just make subtle remarks about how it would have been better if they listened.

Totally unrelated Jeremy, but did you know the SSL cert on https://minops.com/ has expired? (Was interested to see what you were up to these days, which is how I stumbled on it).

Lol yes. It's intentional. It stops spam bots from trying to sign up, since we aren't open for signups yet.

But don't worry, you're not the first to mention it. I suppose I should just fix it and deal with the spam like normal.

I liked the unintended effect of cutting down on spam. I guess a lot of spam bots are written on top of standard libraries that reject bad certs. :)

Also, this was ironically a great way to publicly call someone out for a seemingly bad decision without being cruel about it, so props to you!

Post reply on HN