Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM Cloud was down, as well as their status page

cloud.ibm.com

Post reply on HN
Print view View original

101–110 of 201 posts

Re: IBM Cloud was down, as well as their status page

#101

Post by jedberg »

rezonant wrote: post #99
arkitaip wrote: post #86

Earlier quoted context omitted.

This has to be the most unconventional anti-spam technique I've ever heard about.

> I stumbled on it by accident. I was lazy and let the cert lapse, but then noticed that spam signups basically stopped. One day maybe I'll make a post about it with graphs, although I'm not sure I actually have the data. This is intriguing. I'm going to remember this but I'm too anal about perfect A+ TLS and renewal is already fully automated these days anyway :-\ I wonder if one could setup their TLS stack to get t…

You could probably get the same effect with a self signed cert. Although that wouldn't get you an A+ on TLS. :)

Also, if y'all do this, it probably won't work because the spammers will start ignoring expired certs.

Re: IBM Cloud was down, as well as their status page

#102

Post by all_blue_chucks »

caiobegotti wrote: post #78
all_blue_chucks wrote: post #62

Earlier quoted context omitted.

Never humiliate a coworker in public. Instead say "both options were considered but ultimately it was decided to select option B for reason Y."

My professional experience tells me that the next question will be who decided for B given Y, then you answer it and then you have a target on your SRE back, I'm afraid. Remember that the trickle-down economics works only when the shit hits the fans and what trickles down is not money.

If your company uses post mortems to blame individuals rather than fix processes and tools, you haven't worked in a professional environment.

Re: IBM Cloud was down, as well as their status page

#103

Post by sky_rw »

ComputerGuru wrote: post #53

So what are HNers using IBM Cloud for and where do you see that it has an edge over AWS offerings (where an overlap exists, obviously)? (I figure either you’re in devops and you are putting out fires too busy to read this thread or you’re not and your work is halted because of the incident so you might have time to read and reply ;)

I had originally signed up due to the availability and pricing of bare metal servers and the mixed Windows/Linux server offerings. Their windows server licensing was better than AWS and I didn't want to be on Azure for a variety of reasons.

Currently on them because we have an OpenVPN based infrastructure that is very challenging to migrate.

Lastly the majority of our customers are in the midwest or Texas, and the proximity of their Dallas DC was a huge performance win for us.

Re: IBM Cloud was down, as well as their status page

#104

Post by wmf »

adrr wrote: post #71
Operyl wrote: post #17

Earlier quoted context omitted.

Account managers are texting, but they have no VPN access right now.

It seems all their external network connections are down. I assume people will have to drive to the data centers to fix. I really want to see a post mortem on this outage.

The data centers are staffed 24/7 and out of band is also a thing.

Re: IBM Cloud was down, as well as their status page

#105

Post by rezonant »

jedberg wrote: post #56
caiobegotti wrote: post #24

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

I would mirror the attitude of the person who said no originally. If they are receptive to feedback and clearly want to do better, I would be kind and explain why I had suggested it not be there in the first place and cite this as an example. If they were being adamant or denying it was their fault, I'd probably be really quiet and just make subtle remarks about how it would have been better if they listened.

Sometimes this is the only way :-/ Good to ensure you "get it in writing" when the point is eventually proven in production.

Re: IBM Cloud was down, as well as their status page

#106

Post by dang »

toast0 wrote: post #87
ComputerGuru wrote: post #53

So what are HNers using IBM Cloud for and where do you see that it has an edge over AWS offerings (where an overlap exists, obviously)? (I figure either you’re in devops and you are putting out fires too busy to read this thread or you’re not and your work is halted because of the incident so you might have time to read and reply ;)

We used Softlayer (rebranded to IBM Cloud, and affected by this) at my last job. For the most part, their service pretty much just works; clearly not today. :) We had a couple thousand bare metal servers, and barely used any of their API stuff. As with any facility, there were occasional issues with electrical transfer switches, core router failures, fiber cuts, etc. Stuff happens, but we got pretty good communicatio…

HN ran on a box at Softlayer until early 2018 or so. This makes me think that the title of this post (which was submitted as "IBM Cloud down as well as their status page which looks to be hosted there") could at some point have been "IBM Cloud down as well as their status page which looks to be hosted there as well as the forum where people post these things which also looks to be hosted there".

Re: IBM Cloud was down, as well as their status page

#107

Post by rezonant »

rezonant wrote: post #99
arkitaip wrote: post #86

Earlier quoted context omitted.

This has to be the most unconventional anti-spam technique I've ever heard about.

> I stumbled on it by accident. I was lazy and let the cert lapse, but then noticed that spam signups basically stopped. One day maybe I'll make a post about it with graphs, although I'm not sure I actually have the data. This is intriguing. I'm going to remember this but I'm too anal about perfect A+ TLS and renewal is already fully automated these days anyway :-\ I wonder if one could setup their TLS stack to get t…

My apologies for the limited nesting at the hn nestlimit > You could probably get the same effect with a self signed cert. Although that wouldn't get you an A+ on TLS. :) > Also, if y'all do this, it probably won't work because the spammers will start ignoring expired certs.

Yeah, even if you could find a way to deny the spammers via esoteric configuration, it'll just make them realize they forgot to turn off TLS validation anyway (which is clearly what they meant to do)

Re: IBM Cloud was down, as well as their status page

#108

Post by bigiain »

mbreese wrote: post #75
Operyl wrote: post #14

Earlier quoted context omitted.

Maybe. About 3/4 of all outages get a post mortem. There's 1/4 of the time they refuse to tell us anything.

There will have to be a post mortem on this. The convention is to be as transparent as possible as to what went wrong. This helps to let current customers know that you found the problem, and have put plans in place to make sure it doesn't happen again. The purpose of the signalling here is two fold. 1) If convincing enough (with details), you can keep current customers from moving to a competitor. 2) It also lets ne…

> then they can expect a mass exodus of customers

I wonder if that's a thing that would even cross a typical IBM-ers mind? It might just be me, but I get a very strong smell of "We're IBM! There's nowhere else for you to go!" from them...

Re: IBM Cloud was down, as well as their status page

#109

Post by toast0 »

[deleted] wrote: post #80
whyleyc wrote: post #39

Earlier quoted context omitted.

Is there talk of an ETA for any fix yet?

[deleted]

I managed to capture a traceroute on the lg.softlayer.com from seattle to my home near seattle, that went via London (networklayer to london, then telia back to seattle). lol

Looks like things are getting better now though; looking glass says seattle and dallas can get to me without going to crazy destinations. wdc is still icky.

Re: IBM Cloud was down, as well as their status page

#110

Post by bigiain »

colinbartlett wrote: post #35
Jedd wrote: post #28

Earlier quoted context omitted.

Big fans of StatusGator here. Do you have similar %'s of monitored cloud services that have gone off the air during other providers' outages?

Thanks and that's a GREAT idea for some detailed analysis. I have been trying to make better use of the 5+ years of status page history stashed in the cupboard.

This is something I'd totally read.

(You should headhunt the guy at BackBlaze who does their hard drive stats blog posts, and release this data analysis quarterly!)

Post reply on HN