[deleted]
Is there talk of an ETA for any fix yet?
Never
IBM Cloud was down, as well as their status page
41–50 of 201 posts
Re: IBM Cloud was down, as well as their status page
#42Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…
Don’t bite or embarrass the hand that provides you make-work... Seriously they probably tested it and it worked in theory, just not in practice and now they fix it for reals.
I guess if their DR firedrill assumed their failover router hosting the status page could never go down it would pass, but come on IBM.
Re: IBM Cloud was down, as well as their status page
#43All of Broadcastify's audio servers (hosted with Softlayer in their Dallas datacenter) are completely unreachable and down. I'm going to wait a bit to see if we get a status update, otherwise we'll be spinning up instances on AWS to failover (which will be enormously costly for bandwidth) No status, no nothing, we're in the dark.
Hey. Do you want to shoot me an email, IRC chat, or anything? I can keep you up to date with what I'm hearing from my manager.
After signing the appropriate NDA's and being vetted by IBM legal of course
Re: IBM Cloud was down, as well as their status page
#44Earlier quoted context omitted.
Don’t bite or embarrass the hand that provides you make-work... Seriously they probably tested it and it worked in theory, just not in practice and now they fix it for reals.
I guess if their DR firedrill assumed their failover router hosting the status page could never go down it would pass, but come on IBM.
Seems a common mistake. If I remember right, AWS stumbled over their status page depending on S3 a few years back
Re: IBM Cloud was down, as well as their status page
#45Ah, is this the exception that proves the rule that "no one was ever fired for buying IBM?" Sorry to be glib, I'm sure it's a tough time for people who were sold on their cloud platform and work on it!
Everybody's cloud goes down sometime. The big fail here was hosting their status page on the same infrastructure.
Re: IBM Cloud was down, as well as their status page
#46Earlier quoted context omitted.
Hey. Do you want to shoot me an email, IRC chat, or anything? I can keep you up to date with what I'm hearing from my manager.
After signing the appropriate NDA's and being vetted by IBM legal of course
Everything I'd say would be public knowledge, of course. I don't work for IBM, I'm a user :).
Re: IBM Cloud was down, as well as their status page
#47Re: IBM Cloud was down, as well as their status page
#48Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…
I built a status page for a top cloud provider, and this was question number one from SREs.
Re: IBM Cloud was down, as well as their status page
#49Re: IBM Cloud was down, as well as their status page
#50Earlier quoted context omitted.
Their bare metal cloud offering (SoftLayer acquisition) was actually pretty good whenever I used it about 4 years ago. Wasn’t the most intuitive API or UI but you could get a bare metal server anywhere in the world in a few minutes.
When the wind blows in the right direction. Sometimes, your server would get stuck in provisioning for hours and only get „un-stuck” after creating a support ticket. Which, I kid you not, at one of the previous jobs, wd had automated in our provisioning popeline. Good times. But when it worked, it worked. API was voodoo.
Fixing provisioning based on support tickets might have been automated on their side too :)