Earlier quoted context omitted.
I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.
Joyent us-east-1 rebooted due to operator error
51–60 of 129 posts
Re: Joyent us-east-1 rebooted due to operator error
#52Let this be a lesson to linux admins. Re-alias shutdown -r now into something else on production servers. I once took down access to about 6000 servers because I ran the script to decommission servers on our jump box when I got the SSH windows confused.
Then we got a hp-ux machine in the lab. For some reason, the grace period on that system was in seconds ( http://www.polarhome.com/service/man/generic.php?qf=shutdown... )
System dax shutting down in 5 seconds.
Re: Joyent us-east-1 rebooted due to operator error
#53Earlier quoted context omitted.
I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.
Re: Joyent us-east-1 rebooted due to operator error
#54Earlier quoted context omitted.
You don't intentionally build an automated way to take down all your servers at once. You build a way to automatically perform some mundane standard procedure, like propagating a new firewall rule to all your systems at once. Then you accidentally propagate a rule that blocks all inbound ports. Huh, when I tested locally I didn't notice that. Or you build a way to automatically delete timestamped log files more than…
Once I typed rm -rf logs_ * instead of rm -rf logs_*
Re: Joyent us-east-1 rebooted due to operator error
#55Looks like the janitor needed somewhere to plug in the vacuum cleaner again...
I assume bash.org?
http://thedailywtf.com/Articles/I-Didnt-Do-Anything.aspx
Unintentional Mishap while Contractor Unplugs X to fix/maintain Y is a relatively common theme on their list of horror stories.
edit: I think he might actually have meant this one: http://thedailywtf.com/Articles/I-Told-You-So.aspx
Re: Joyent us-east-1 rebooted due to operator error
#56Re: Joyent us-east-1 rebooted due to operator error
#57Earlier quoted context omitted.
I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.
Re: Joyent us-east-1 rebooted due to operator error
#58Earlier quoted context omitted.
I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.
Re: Joyent us-east-1 rebooted due to operator error
#59Joyent's messaging about "we're cloud, but with perfect uptime" was always broken. It's mildly gross that the current messaging sounds like they're throwing a sysadmin under the bus. If fat fingers can down a data center, that's an engineering problem. I care about an object store that never loses data and an API that always has an answer for me, even if it's saying things that I don't want to hear. 99.999 sounds stu…
Sure blame on the engineers. You give power, people use it badly blame the engineer for giving too much power. You don't give enough power sysadmins/users bitch and yell why don't we have enough power, we're not children. Its always the engineer fault. :(
Re: Joyent us-east-1 rebooted due to operator error
#60Earlier quoted context omitted.
I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.