Live data from Hacker News

Joyent us-east-1 rebooted due to operator error

help.joyent.com

51–60 of 129 posts

Re: Joyent us-east-1 rebooted due to operator error

#51

Earlier quoted context omitted.

I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

really? what can cost that much ?

Re: Joyent us-east-1 rebooted due to operator error

#52

Let this be a lesson to linux admins. Re-alias shutdown -r now into something else on production servers. I once took down access to about 6000 servers because I ran the script to decommission servers on our jump box when I got the SSH windows confused.

At one point, I worked in a computer lab that was mostly Ultrix machines. The shutdown grace period was specified in minutes ( http://www.polarhome.com/service/man/generic.php?qf=shutdown... )

Then we got a hp-ux machine in the lab. For some reason, the grace period on that system was in seconds ( http://www.polarhome.com/service/man/generic.php?qf=shutdown... )

System dax shutting down in 5 seconds.

Re: Joyent us-east-1 rebooted due to operator error

#53

Earlier quoted context omitted.

I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

Wow. Did this gold-plated B-2 bomber still fly after the coffee incident?

Re: Joyent us-east-1 rebooted due to operator error

#54
post #39

Earlier quoted context omitted.

You don't intentionally build an automated way to take down all your servers at once. You build a way to automatically perform some mundane standard procedure, like propagating a new firewall rule to all your systems at once. Then you accidentally propagate a rule that blocks all inbound ports. Huh, when I tested locally I didn't notice that. Or you build a way to automatically delete timestamped log files more than…

Once I typed rm -rf logs_ * instead of rm -rf logs_*

I have nightmares about such things.

Re: Joyent us-east-1 rebooted due to operator error

#55
post #20

Looks like the janitor needed somewhere to plug in the vacuum cleaner again...

I assume bash.org?

He might be referring to The daily WTF (worse than failure):

http://thedailywtf.com/Articles/I-Didnt-Do-Anything.aspx

Unintentional Mishap while Contractor Unplugs X to fix/maintain Y is a relatively common theme on their list of horror stories.

edit: I think he might actually have meant this one: http://thedailywtf.com/Articles/I-Told-You-So.aspx

Re: Joyent us-east-1 rebooted due to operator error

#56

Looks like the janitor needed somewhere to plug in the vacuum cleaner again...

Not even just the plug. I've had outages from bits flipped simply by the static electricity generated when vacuuming near servers.

You have carpets in your server room??

Re: Joyent us-east-1 rebooted due to operator error

#57

Earlier quoted context omitted.

I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

You know you could build five space shuttles with ten zeros, right? Are we talking dollars or Yen?

Re: Joyent us-east-1 rebooted due to operator error

#58

Earlier quoted context omitted.

I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

Seems like not allowing food or drink near a device worth over 10 zeroes would be a no-brainer, but hindsight is tricky like that.

Re: Joyent us-east-1 rebooted due to operator error

#59
post #50
post #2

Joyent's messaging about "we're cloud, but with perfect uptime" was always broken. It's mildly gross that the current messaging sounds like they're throwing a sysadmin under the bus. If fat fingers can down a data center, that's an engineering problem. I care about an object store that never loses data and an API that always has an answer for me, even if it's saying things that I don't want to hear. 99.999 sounds stu…

Sure blame on the engineers. You give power, people use it badly blame the engineer for giving too much power. You don't give enough power sysadmins/users bitch and yell why don't we have enough power, we're not children. Its always the engineer fault. :(

It's a combined fault. Clearly the operator made a mistake, but the system shouldn't have let such a calamitous operation take place without at least three levels of "Are you sure" (or something smarter like "Confirm how many servers would you like to reboot:") before it lets you take down thousands of servers.

Re: Joyent us-east-1 rebooted due to operator error

#60

Earlier quoted context omitted.

I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.

My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.

if something worth over 10 zeroes can be destroyed with a coffee spill, i would say it had it coming
Post reply on HN