Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM Cloud was down, as well as their status page

cloud.ibm.com

Post reply on HN
Print view View original

141–150 of 201 posts

Re: IBM Cloud was down, as well as their status page

#141

Post by epc »

caiobegotti wrote: post #24

Honest slightly cynical question: most probably someone inside the responsible team said some day that it would be very stupid to host the status page inside the same infrastructure being monitored, but they were probably ignored... what should that person do now? Say "toldya!" out loud in the postmortem meeting or simply shut up and move on because reality is that we are hired to do some stupid task and not to think…

I haven't been at IBM since 2001, but when I was there any suggestion like this would have been beaten down by multiple layers of the big grey cloud for even intimating that such a visible, key piece of IBM marketing material should be on a third party service.

Re: IBM Cloud was down, as well as their status page

#142

Post by bashinator »

rad_gruchalski wrote: post #31
twalla wrote: post #25

Earlier quoted context omitted.

Their bare metal cloud offering (SoftLayer acquisition) was actually pretty good whenever I used it about 4 years ago. Wasn’t the most intuitive API or UI but you could get a bare metal server anywhere in the world in a few minutes.

When the wind blows in the right direction. Sometimes, your server would get stuck in provisioning for hours and only get „un-stuck” after creating a support ticket. Which, I kid you not, at one of the previous jobs, wd had automated in our provisioning popeline. Good times. But when it worked, it worked. API was voodoo.

I just discovered this today:

    aws support create-case \
        --subject "not working" \
        --communication-body file://description.txt

Re: IBM Cloud was down, as well as their status page

#144

Post by matthijs »

Scoundreller wrote: post #133
toast0 wrote: post #124

Earlier quoted context omitted.

> There were a lot of sad panda hats when you guys started to transition away. For us as well. It was so nice to have things work one day and the next and the next, although I guess they wouldn't have worked today. Favorite firefighting moment was when wdc lost half the fiber in ~ 2014, and we had to move all of our traffic out, so that there was capacity. Our guy asked why we had to move? and your guy said something…

Dunno if I should realize who toast0 is, but what service were/are you running?

Probably WhatsApp, they moved from FreeBSD to Linux and were on SL before the Facebook acquisition.

Re: IBM Cloud was down, as well as their status page

#145

Post by toast0 »

Operyl wrote: post #90
colinbartlett wrote: post #74

Earlier quoted context omitted.

Do you actually have data on that or are you conjecturing? Because I would really love to see data about that if it exists somewhere.

I'm talking from experience. Most things do get post mortems, but there's a lot of crap they also don't give us post mortems for "because customer data." It's my number 1 complaint, and I fight with managers about this all the time. We have a ton of hypervisor problems, and a lot of networking issues (generally over private network) and they tend to get very very secretive about it.

I didn't use their hypervisors, but I've had a lot of experience troubleshooting their networks. They've gotten a lot better at proactive monitoring, but we used to occassionally find some private networking paths that were having trouble, and until we narrowed it down, it was hard to find. (I dunno, I guess you can't just ask all the routers if there are any ports with errors, but sure enough, when they found the right port, there was usually a huge error count, or something)

The key thing is each IP 5-tuple (peerA, peerB, protocol, portA, portB) will always take the same path over their network (most likely a different path for return packets, when A and B are switched), so in order to properly probe, you need to probe on a lot of of port combos, and once you find a broken combo, you need to run MTR on those ports, so you can give them the MTR that shows the issue.

Or, if you can, have your internode protocol run on multiple connections and drop connections that are showing issues, and let a different customer file the tickets :)

(email is in my profile if you want to discuss)

Re: IBM Cloud was down, as well as their status page

#146

Post by toast0 »

ck2 wrote: post #115

even weather.com was down but someone broke ebay too Fastly error: unknown domain: www.ebay.com. Please check that this domain has been added to a service.

weather.com makes sense. IBM bought the weather channel a while ago, hosting is likely tied to IBM Cloud at this point (although it looks like it's fronted by Akamai)

Re: IBM Cloud was down, as well as their status page

#147

Post by jonfw »

voz_ wrote: post #11

I generally do everything on AWS or GCP, with a little Azure sometimes for personal projects. In what world does IBM beat one of those three in anything? Generally curious - how they are able to stay competitive?

They've got the only real managed Openshift option right now, and their managed Kubernetes services is really great and seamless IMO.

Re: IBM Cloud was down, as well as their status page

#148

Post by dsmcr »

toast0 wrote: post #139
Scoundreller wrote: post #133

Earlier quoted context omitted.

Dunno if I should realize who toast0 is, but what service were/are you running?

I don't think you are expected to know who I am :) I omitted the service on purpose, but you my email is on my profile if you want to know. Apparently it was enough information for dsmcr to properly id the service though; not enough for nixgeek though, I think.

Sorry, didn't mean to put you on blast like that.

Re: IBM Cloud was down, as well as their status page

#149

Post by jonfw »

ethbro wrote: post #129
all_blue_chucks wrote: post #62

Earlier quoted context omitted.

Never humiliate a coworker in public. Instead say "both options were considered but ultimately it was decided to select option B for reason Y."

What if Y = ignorance?

Where are all of these managers out there running companies making decisions with literally no thought whatsoever? I've literally never seen them- I almost exclusively work with rational human beings who are able to justify decisions, and the few who aren't haven't been afforded any real power.
Post reply on HN