Live data from Hacker News

Joyent us-east-1 rebooted due to operator error

help.joyent.com

101–110 of 129 posts

Re: Joyent us-east-1 rebooted due to operator error

#101
post #65

Earlier quoted context omitted.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

supposedly 0.0000000001 billions of $ shit happens, design for the worst.

"If you reach for a star, and come up with a handful of mud..."

Re: Joyent us-east-1 rebooted due to operator error

#102

Earlier quoted context omitted.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

You know you could build five space shuttles with ten zeros, right? Are we talking dollars or Yen?

Why are you using a throwaway account? Ohh, I just saw the "dollars or Yen" remark. TIL we use throwaway accounts for the times we feel like being assholes, so the non-elites can't track it back to our physical neuroprocessors.

Re: Joyent us-east-1 rebooted due to operator error

#103
post #60

Earlier quoted context omitted.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

if something worth over 10 zeroes can be destroyed with a coffee spill, i would say it had it coming

[deleted]

Re: Joyent us-east-1 rebooted due to operator error

#104
post #22

Earlier quoted context omitted.

> sounds like they're throwing a sysadmin under the bus at least they didn't name the operator in question...

Our internal culture is such that everyone on the team would rather be blamed for something than accuse someone else of doing it. That's shitty, and not something you do to someone. You fix the problem and then you move on. If it makes you happy, blame me - I don't mind.

[deleted]

Re: Joyent us-east-1 rebooted due to operator error

#105

Earlier quoted context omitted.

I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.

It's operations. You fuck up, you suck it up, you fix it, then (and this is the important part) you prevent it from ever happening again. Feeling like shit for bringing something down is a good way to give yourself depression, given how often you will screw the pooch with root. In the same vein, anybody who says they'd fire the operator without any qualification on that remark should be given a wide berth. People ten…

[deleted]

Re: Joyent us-east-1 rebooted due to operator error

#106

Earlier quoted context omitted.

You know you could build five space shuttles with ten zeros, right? Are we talking dollars or Yen?

Why are you using a throwaway account? Ohh, I just saw the "dollars or Yen" remark. TIL we use throwaway accounts for the times we feel like being assholes, so the non-elites can't track it back to our physical neuroprocessors.

I'm not sure why I'm dignifying this Reddit drivel with a response, but my karma and account age should be your hint that you're barking up the wrong tree.

Re: Joyent us-east-1 rebooted due to operator error

#107

Earlier quoted context omitted.

Why are you using a throwaway account? Ohh, I just saw the "dollars or Yen" remark. TIL we use throwaway accounts for the times we feel like being assholes, so the non-elites can't track it back to our physical neuroprocessors.

I'm not sure why I'm dignifying this Reddit drivel with a response, but my karma and account age should be your hint that you're barking up the wrong tree.

"check my previous responses and my credit score for how you should treat me" ohh what an old-man response. It's too bad the Imgur "downvote everything they ever posted" script doesn't work here on HN, now isn't it?

My account's karma and history exceed your account's on this site, and even worse, this individual comment bears more value than yours! Ooh burn!

Re: Joyent us-east-1 rebooted due to operator error

#108
post #17
post #14

Earlier quoted context omitted.

So don't give anyone root on an entire data center .

Is this like Captain Planet? It's a bit exceptional to divide access servers of similar type between administrators such that individuals have full access to a portion of the fleet. Do they meet up and put their rings together to roll out updates? What if one of them goes on vacation?

Actually, capabilities makes it trivial to lock down things like shutdown for admin accounts. A script can do the shutdown instead in a more controlled and less error-prone fashion. Same for network device updates. Abstraction.

Re: Joyent us-east-1 rebooted due to operator error

#109
post #87

Earlier quoted context omitted.

This comes down to operational philosophy, in the end. The point you're dancing around is whether the system should permit grave actions that don't make any sense when you're designing the system. By the rules, every single system on a commercial aircraft has a circuit breaker. Pilots make the "what if X catches on fire?" case, which is actually pretty compelling. However, that also means there are several switches o…

I tend to agree with you, with the caveat that you can't have this philosophy and sell your customers 99.999% uptime[0]. [0] http://www.joyent.com/products/compute-service/features/linu...

They're not. That's a statement of what customers have enjoyed up until now. The actual SLA simply states what refund you get for each hour of downtime.

Re: Joyent us-east-1 rebooted due to operator error

#110
post #67

Earlier quoted context omitted.

How many non-zero integers were in the price of the device, though? My computer is also worth more than 10 zeros.

Please keep in mind that "price" means "how many dollars other humans are willing to trade for it right now"; not necessarily any concrete evaluation of the device's functionality compared to a human competitor or human operator...

DOWNVOTE FOR TRUTH
Post reply on HN