Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Joyent us-east-1 rebooted due to operator error

help.joyent.com

Post reply on HN
Print view View original

121–129 of 129 posts

Re: Joyent us-east-1 rebooted due to operator error

#121

Post by 4ad »

berns wrote: post #70
alrs wrote: post #2

Joyent's messaging about "we're cloud, but with perfect uptime" was always broken. It's mildly gross that the current messaging sounds like they're throwing a sysadmin under the bus. If fat fingers can down a data center, that's an engineering problem. I care about an object store that never loses data and an API that always has an answer for me, even if it's saying things that I don't want to hear. 99.999 sounds stu…

Joyent's marketing is not the most transparent. They haven't updated AWS prices in their pricing page since AWS lowered their prices two months ago.

What?

Joyent doesn't use AWS.

Re: Joyent us-east-1 rebooted due to operator error

#122

Post by Gigablah »

disillusioned wrote: post #118
stephengillie wrote: post #107

Earlier quoted context omitted.

"check my previous responses and my credit score for how you should treat me" ohh what an old-man response. It's too bad the Imgur "downvote everything they ever posted" script doesn't work here on HN, now isn't it? My account's karma and history exceed your account's on this site, and even worse, this individual comment bears more value than yours! Ooh burn!

Haha, this is a pretty spectacular amount of cognitive dissonance you're demonstrating here. Let's post-mortem this lunacy: 1) You misinterpret jsmthrowaway's initial comment as vaguely racist (or something), notice the word "throwaway" in his user name and get really excited that you can stand on your high horse and call him out for hiding behind the shield of internet anonymity when he wants to be a (you think) rac…

It seems he has a bit of misplaced sensitivity when it comes to the Japanese: https://news.ycombinator.com/item?id=7555232

Re: Joyent us-east-1 rebooted due to operator error

#123

Post by mcherm »

jsmthrowaway wrote: post #73
alrs wrote: post #63

Earlier quoted context omitted.

Systems engineers, software engineers, architects, whatever. We're all in the same gang. My point is that the problem in this case is likely the system's design, not one engineer's typing abilities.

This comes down to operational philosophy, in the end. The point you're dancing around is whether the system should permit grave actions that don't make any sense when you're designing the system. By the rules, every single system on a commercial aircraft has a circuit breaker. Pilots make the "what if X catches on fire?" case, which is actually pretty compelling. However, that also means there are several switches o…

...and the operations version of that is that all normal operations are performed under restricted permissions that cannot "do anything", while the full "do anything" permissions are only broken out during a major crisis.

Such an approach would have prevented this incident where "normal" operations were being performed and accidentally ALL the servers were rebooted at once.

Re: Joyent us-east-1 rebooted due to operator error

#124

Post by stephengillie »

disillusioned wrote: post #118
stephengillie wrote: post #107

Earlier quoted context omitted.

"check my previous responses and my credit score for how you should treat me" ohh what an old-man response. It's too bad the Imgur "downvote everything they ever posted" script doesn't work here on HN, now isn't it? My account's karma and history exceed your account's on this site, and even worse, this individual comment bears more value than yours! Ooh burn!

Haha, this is a pretty spectacular amount of cognitive dissonance you're demonstrating here. Let's post-mortem this lunacy: 1) You misinterpret jsmthrowaway's initial comment as vaguely racist (or something), notice the word "throwaway" in his user name and get really excited that you can stand on your high horse and call him out for hiding behind the shield of internet anonymity when he wants to be a (you think) rac…

I was wildly drunk when posting this. I didn't remember doing so until I checked HN just now.

Re: Joyent us-east-1 rebooted due to operator error

#125

Post by SEJeff »

SEJeff wrote: post #3

As I've always said, "You can never protect a system from a stupid person with root". You can limit carnage and mitigate this type of thing, but you can't fully protect against sysadmins doing dumb things (unless you just hire great sysadmins)

Wow HN seriously!? I never once pretended that I'm able to hire people who don't make mistakes, only that you can't protect systems from administrators who mess up.

Get a grip people.

Re: Joyent us-east-1 rebooted due to operator error

#126

Post by macavity23 »

stephengillie wrote: post #113
myrandomcomment wrote: post #66

Earlier quoted context omitted.

Back in the day we used to say there are 2 types of network engineers, those that have dropped a backbone and those that will drop a backbone.

Your management is failing your newer engineers, if this is still the case.

Nonsense. Someone has to be operating at the sharp end of the enable prompt, and sooner or later it'll be 0330 and that person will type Ethernet0 when they meant Ethernet1, whatever management you have in place.

When that happens, you do just what Joyent did here: you send out an embarrassed email to customers, everyone else in the ops team gets a few cheap laughs at the miscreant's expense, you have a meeting about it, discuss lessons learned, and you move on.

Everyone screws up. Everything goes down once in a while. This is why you build in redundancy at every level.

Re: Joyent us-east-1 rebooted due to operator error

#127

Post by elijahwright »

alrs wrote: post #92
jsmthrowaway wrote: post #89

Earlier quoted context omitted.

I disagree wholeheartedly. Your operational philosophy complements your SLA goals, it doesn't force them.

I can't figure out how your comment that "understanding mistakes will happen" is compatible with 99.999% uptime. I'm of the opinion that 99.999% for an individual instance isn't particularly achievable in a commodity hosting environment. That kind of uptime doesn't leave much room for the mistakes that you and I both anticipate. I do think that 99.999% is doable for a properly distributed whole-system across multiple…

Always ask how numbers like that are computed.

Re: Joyent us-east-1 rebooted due to operator error

#128

Post by saganus »

saganus wrote: post #20
jordanthoms wrote: post #15

Looks like the janitor needed somewhere to plug in the vacuum cleaner again...

I assume bash.org?

And I get 2 downvotes for this? really? downvoters care to explain why, just for asking if it was a reference from bash? Wow... Edit: Thanks to the other 2 posters who provided alternative sources. You learn by asking, no? or at least some of us do..

Re: Joyent us-east-1 rebooted due to operator error

#129

Post by chris_wot »

stephengillie wrote: post #107
jsmthrowaway wrote: post #106

Earlier quoted context omitted.

I'm not sure why I'm dignifying this Reddit drivel with a response, but my karma and account age should be your hint that you're barking up the wrong tree.

"check my previous responses and my credit score for how you should treat me" ohh what an old-man response. It's too bad the Imgur "downvote everything they ever posted" script doesn't work here on HN, now isn't it? My account's karma and history exceed your account's on this site, and even worse, this individual comment bears more value than yours! Ooh burn!

Uh, you just called him out for using a throwaway account, now age and experience means nothing? Go away, troll!
Post reply on HN