Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Joyent us-east-1 rebooted due to operator error

help.joyent.com

Post reply on HN
Print view View original

91–100 of 129 posts

Re: Joyent us-east-1 rebooted due to operator error

#91

Post by devinegan »

rincebrain wrote: post #78
devinegan wrote: post #43

Joyent has been having some serious issues over the past month or two. I am not sure if it is growing pains, bad luck or what is happening, but we had already lost faith and trust in their Cloud prior to today. This is the nail in the coffin from our perspective. Moving on...

Howso?

Thanks for asking rather than just down-voting. I wanted others to know that this isn't isolated. We have been having issues with their service for a few months now. They never know when there is a problem with hardware, for instance. Joyent support will gladly tell you everything is fine. After you insist, and insist they will actually have someone look at the underlying infrastructure. Eventually they will acknowledge the problem and fix it (maybe). I believe the monitoring and reporting for their team is flawed or incomplete which leads to more downtime of affected systems. Just one observation, but we have had three incidents over the past month and a half. Two within a week of each other.

Re: Joyent us-east-1 rebooted due to operator error

#92

Post by alrs »

jsmthrowaway wrote: post #89
alrs wrote: post #87

Earlier quoted context omitted.

I tend to agree with you, with the caveat that you can't have this philosophy and sell your customers 99.999% uptime[0]. [0] http://www.joyent.com/products/compute-service/features/linu...

I disagree wholeheartedly. Your operational philosophy complements your SLA goals, it doesn't force them.

I can't figure out how your comment that "understanding mistakes will happen" is compatible with 99.999% uptime.

I'm of the opinion that 99.999% for an individual instance isn't particularly achievable in a commodity hosting environment. That kind of uptime doesn't leave much room for the mistakes that you and I both anticipate.

I do think that 99.999% is doable for a properly distributed whole-system across multiple geographically-dispersed datacenters.

I think Joyent has gone wrong in promoting individual instance reliability.

Re: Joyent us-east-1 rebooted due to operator error

#93

Post by mpyne »

pdeva1 wrote: post #60
socceroos wrote: post #47

Earlier quoted context omitted.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

if something worth over 10 zeroes can be destroyed with a coffee spill, i would say it had it coming

As a one-time newish officer who used to be in charge of things with many zeroes, I'd be inclined to agree.

Re: Joyent us-east-1 rebooted due to operator error

#94

Post by linker3000 »

hack_edu wrote: post #38
jordanthoms wrote: post #15

Looks like the janitor needed somewhere to plug in the vacuum cleaner again...

Not even just the plug. I've had outages from bits flipped simply by the static electricity generated when vacuuming near servers.

5W walkie talkies in a big sports complex with the RF getting into the keyboard controllers and acting like a maniac was punching the keyboard - would eventually hang the servers.

Fix: Replace cheapened-keyboards-with-mylar-film-(not)-screening with older models that had a full metal cage around the keyboard assembly.

Re: Joyent us-east-1 rebooted due to operator error

#95

Post by M2Ys4U »

sokoloff wrote: post #71
elijahwright wrote: post #30

Earlier quoted context omitted.

Our internal culture is such that everyone on the team would rather be blamed for something than accuse someone else of doing it. That's shitty, and not something you do to someone. You fix the problem and then you move on. If it makes you happy, blame me - I don't mind.

At my $DAYJOB, we are always careful to figure out exactly what happened, including by whom. It's not to assign personal blame, but I believe it's critical that everyone agrees on the facts (who, what, when, where, and [if possible] why). Response and conversation is always focused on "how do we prevent this in the future?", not on punishing whoever was involved in the past. IOW, I agree with I believe is your intent…

I think there's a difference in how you approach this with an internal-facing view and an external one.

Internally, You're right. But externally the company fucked up, not the individual.

Re: Joyent us-east-1 rebooted due to operator error

#96

Post by sokoloff »

M2Ys4U wrote: post #95
sokoloff wrote: post #71

Earlier quoted context omitted.

At my $DAYJOB, we are always careful to figure out exactly what happened, including by whom. It's not to assign personal blame, but I believe it's critical that everyone agrees on the facts (who, what, when, where, and [if possible] why). Response and conversation is always focused on "how do we prevent this in the future?", not on punishing whoever was involved in the past. IOW, I agree with I believe is your intent…

I think there's a difference in how you approach this with an internal-facing view and an external one. Internally, You're right. But externally the company fucked up, not the individual.

100% agree, and it is my oversight to not draw that distinction more clearly. We have the luxury (so far) of only reporting internally.

Re: Joyent us-east-1 rebooted due to operator error

#97

Post by dsberkholz »

jsmthrowaway wrote: post #57
socceroos wrote: post #47

Earlier quoted context omitted.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

You know you could build five space shuttles with ten zeros, right? Are we talking dollars or Yen?

Must be counting the two after the decimal point. =)

Re: Joyent us-east-1 rebooted due to operator error

#98

Post by bcantrill »

devinegan wrote: post #91
rincebrain wrote: post #78

Earlier quoted context omitted.

Howso?

Thanks for asking rather than just down-voting. I wanted others to know that this isn't isolated. We have been having issues with their service for a few months now. They never know when there is a problem with hardware, for instance. Joyent support will gladly tell you everything is fine. After you insist, and insist they will actually have someone look at the underlying infrastructure. Eventually they will acknowle…

I'm sorry to hear about your experience; we pride ourselves on being able to root-cause problems regardless of where they might be in the stack, but it sounds like your problem didn't get properly escalated. If you want to reach out to me privately (my HN login at acm.org), we can try to figure out what happened here -- with my apologies again for the subpar experience.

Re: Joyent us-east-1 rebooted due to operator error

#99

Post by [deleted] »

myrandomcomment wrote: post #66
jsmthrowaway wrote: post #48

Earlier quoted context omitted.

It's operations. You fuck up, you suck it up, you fix it, then (and this is the important part) you prevent it from ever happening again. Feeling like shit for bringing something down is a good way to give yourself depression, given how often you will screw the pooch with root. In the same vein, anybody who says they'd fire the operator without any qualification on that remark should be given a wide berth. People ten…

Back in the day we used to say there are 2 types of network engineers, those that have dropped a backbone and those that will drop a backbone.

[deleted]

Re: Joyent us-east-1 rebooted due to operator error

#100

Post by stephengillie »

malchow wrote: post #67
socceroos wrote: post #47

Earlier quoted context omitted.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

How many non-zero integers were in the price of the device, though? My computer is also worth more than 10 zeros.

Please keep in mind that "price" means "how many dollars other humans are willing to trade for it right now"; not necessarily any concrete evaluation of the device's functionality compared to a human competitor or human operator...
Post reply on HN