Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM Cloud was down, as well as their status page

cloud.ibm.com

Post reply on HN
Print view View original

51–60 of 201 posts

Re: IBM Cloud was down, as well as their status page

#51

Post by isclever »

caiobegotti wrote: post #24

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

It happens a lot, when you have so much infrastructure and redundancy you think it is too big to fail. Then you lose S3 in US-East1 and break everything.

https://www.theregister.com/2017/03/01/aws_s3_outage/

Re: IBM Cloud was down, as well as their status page

#52

Post by Fordec »

I remember I was at an IBM sponsored hackathon around 2015 where it was a requirement to use Bluemix. Over the course of the weekend the service went down for hours 3 times.

Literally this morning I was wondering what ever happened to it, like did it die a quiet death? Oh it rebranded to IBM cloud in 2017. Now this news.

I think there's an eponymous law named for this sort of thing.

Re: IBM Cloud was down, as well as their status page

#53

Post by ComputerGuru »

So what are HNers using IBM Cloud for and where do you see that it has an edge over AWS offerings (where an overlap exists, obviously)?

(I figure either you’re in devops and you are putting out fires too busy to read this thread or you’re not and your work is halted because of the incident so you might have time to read and reply ;)

Re: IBM Cloud was down, as well as their status page

#55

Post by mbreese »

detaro wrote: post #44
acruns wrote: post #42

Earlier quoted context omitted.

I guess if their DR firedrill assumed their failover router hosting the status page could never go down it would pass, but come on IBM.

Seems a common mistake. If I remember right, AWS stumbled over their status page depending on S3 a few years back

I view this as growing pains that everyone has to learn the hard way. The bigger question is will they learn the lesson and how will they fix this for the next time?

(Because there is always a next time)

Re: IBM Cloud was down, as well as their status page

#56

Post by jedberg »

caiobegotti wrote: post #24

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

I would mirror the attitude of the person who said no originally.

If they are receptive to feedback and clearly want to do better, I would be kind and explain why I had suggested it not be there in the first place and cite this as an example.

If they were being adamant or denying it was their fault, I'd probably be really quiet and just make subtle remarks about how it would have been better if they listened.

Re: IBM Cloud was down, as well as their status page

#57

Post by manquer »

caiobegotti wrote: post #24

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

If such people raised concerns and they had been overridden they way you describe they would sadly have left long ago.

It is not that companies become consciously malicious or are incompetent to start with, it becomes a vicious cycle, as more and more poor management and engineering talent join, the good ones leave, and the cycle continues.

Acquisitions and merge stave off the slow slide into irrelevance for a while, till the best of the new guys leave too. Systemic cultural changes is very very hard to achieve in large organizations.

Re: IBM Cloud was down, as well as their status page

#58

Post by snowwrestler »

caiobegotti wrote: post #24

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

Best thing to do now is to point DNS at the backup status page they discreetly set up on a free-tier EC2 server back when they got ignored...

Re: IBM Cloud was down, as well as their status page

#59

Post by vsareto »

caiobegotti wrote: post #24

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

Honestly it's small compared to everything else. I'd rather leave than do told-ya-so though and put the story in the exit interview or reason for leaving.

Re: IBM Cloud was down, as well as their status page

#60

Post by sky_rw »

The most infuriating thing about this is the ZERO communication coming out of IBM Cloud. No emails. No updates to twitter. Status page down. Support lines clogged.

At least give me something I can point my customers at to show them this is not due to my incompetence.

Post reply on HN