Earlier quoted context omitted.
supposedly 0.0000000001 billions of $ shit happens, design for the worst.
"If you reach for a star, and come up with a handful of mud..."
DOWNVOTE FOR TRUTH
Joyent us-east-1 rebooted due to operator error
111–120 of 129 posts
Re: Joyent us-east-1 rebooted due to operator error
#112Earlier quoted context omitted.
Sorry - are you telling us you had to reboot all nodes because you swapped a router out? Sounds like you need a network engineer.
And I'm being downvoted for that? Seriously? In 13 years of networking I have never once had to reload machine to help with OSPF or BGP convergence. Good networking architecture and planning should mitigate anything other than a couple of minute outage. No routing change should ever require a reload of a server or end node.
You're assuming my management has been paying for good networking architecture for the past dozen years.
Re: Joyent us-east-1 rebooted due to operator error
#113Earlier quoted context omitted.
It's operations. You fuck up, you suck it up, you fix it, then (and this is the important part) you prevent it from ever happening again. Feeling like shit for bringing something down is a good way to give yourself depression, given how often you will screw the pooch with root. In the same vein, anybody who says they'd fire the operator without any qualification on that remark should be given a wide berth. People ten…
Back in the day we used to say there are 2 types of network engineers, those that have dropped a backbone and those that will drop a backbone.
Your management is failing your newer engineers, if this is still the case.
Re: Joyent us-east-1 rebooted due to operator error
#114Earlier quoted context omitted.
And I'm being downvoted for that? Seriously? In 13 years of networking I have never once had to reload machine to help with OSPF or BGP convergence. Good networking architecture and planning should mitigate anything other than a couple of minute outage. No routing change should ever require a reload of a server or end node.
I believe you were down voted not for what you said, but the way you have said it. I've been down voted several times for (what I see) as relatively minor remarks. The HN readers are a sensitive bunch...
Those who are still posting on HN are orders of magnitude more sensitive than those who post on Imgur. The communities are similar-size, yet Imguraffes are much, much more accepting of my comments. What merits a handful of upvotes there brings a downvote or two on this site.
Re: Joyent us-east-1 rebooted due to operator error
#115Earlier quoted context omitted.
And I'm being downvoted for that? Seriously? In 13 years of networking I have never once had to reload machine to help with OSPF or BGP convergence. Good networking architecture and planning should mitigate anything other than a couple of minute outage. No routing change should ever require a reload of a server or end node.
You're assuming my management has been paying for good networking architecture for the past dozen years.
I believe it. Networking is seen as a commodity now. It's transparent until it fails. There's a whole lot of technical debt lurking out there. I personally have seen the dark shadow of spanning tree suck the light out of DevOp engineers eyes.
Re: Joyent us-east-1 rebooted due to operator error
#116Earlier quoted context omitted.
I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.
It's operations. You fuck up, you suck it up, you fix it, then (and this is the important part) you prevent it from ever happening again. Feeling like shit for bringing something down is a good way to give yourself depression, given how often you will screw the pooch with root. In the same vein, anybody who says they'd fire the operator without any qualification on that remark should be given a wide berth. People ten…
I've seen generally brilliant people be bit by bad process. The worst example was an important hard drive being wiped thanks to a lack of labeling, obviously taking a production server down with it.
Other things that have caused outages: lack of power capacity planning, unplugging an unrelated test server from the network (go go gadget BGP), cascading backup power failure, building maintenance taking down AC units, expensive equipment caching ARP replies indefinitely… the list goes on.
I had my own fun fuckup too. I learned SQL on PostgreSQL, and had to fix a problem with logged data in a MySQL database. Not trusting myself, I typed "BEGIN;" to enter a transaction, ran my update, and queried the table to check my results. I noticed my update did more than I expected, so I entered "ROLLBACK;" only to learn that MyISAM tables don't actually implement transactions.
Thankfully, in this case it turned out to be possible to undo the damage, but talk about a heart-stopping moment!
Shit happens. You deal with it, then do what you can to keep it from happening again. I've learned to respect early morning change windows as a way to limit damage caused by mistakes.
Re: Joyent us-east-1 rebooted due to operator error
#117Let this be a lesson to linux admins. Re-alias shutdown -r now into something else on production servers. I once took down access to about 6000 servers because I ran the script to decommission servers on our jump box when I got the SSH windows confused.
Might I suggest molly-guard: https://packages.debian.org/unstable/admin/molly-guard
Cheers for this! Would have saved me so much grief before. Now going around and installing it on the servers I manage (fortunately nothing mission critical, but many remote).
Re: Joyent us-east-1 rebooted due to operator error
#118Earlier quoted context omitted.
I'm not sure why I'm dignifying this Reddit drivel with a response, but my karma and account age should be your hint that you're barking up the wrong tree.
"check my previous responses and my credit score for how you should treat me" ohh what an old-man response. It's too bad the Imgur "downvote everything they ever posted" script doesn't work here on HN, now isn't it? My account's karma and history exceed your account's on this site, and even worse, this individual comment bears more value than yours! Ooh burn!
Haha, this is a pretty spectacular amount of cognitive dissonance you're demonstrating here.
Let's post-mortem this lunacy:
1) You misinterpret jsmthrowaway's initial comment as vaguely racist (or something), notice the word "throwaway" in his user name and get really excited that you can stand on your high horse and call him out for hiding behind the shield of internet anonymity when he wants to be a (you think) racist idiot. Even though his comment is a legitimate currency conversion remark. See: https://www.google.com/search?q=1+usd+in+yen
2) He explains that clearly you're mistaken (which, really, seriously, what a kneejerk response from someone who just wanted to show how clever they were, calling out an "asshole") and further explains his account isn't even a "throwaway" in the traditional, trolly sense of the word, citing his account age and the fact that he regularly actively posts to the account.
3) You, caught perhaps in a moment of clarity, though I think I give you too much credit here, realize you were too eager to pounce on the "asshole" for his "Yen" remark, and perhaps you misread it. Your latent erection fading, you counter by explaining that your history and karma are even _more_ impressive, somehow completely avoiding taking responsibility for a completely nonsensical leap in logic and accusation of wrongdoing, while doubling down on your cognitive dissonance.
4) The Aristocrats!
Re: Joyent us-east-1 rebooted due to operator error
#119Looks like the janitor needed somewhere to plug in the vacuum cleaner again...
I assume bash.org?
It's a truly ancient anecdote; it probably predates the Internet.
The first example in RISKS is in 1994: http://catless.ncl.ac.uk/Risks/15.59.html#subj3.1 but the canonical version of the story is in a Cape Town hospital in 1996: http://web.archive.org/web/20040624065333/http://www.legends...
Re: Joyent us-east-1 rebooted due to operator error
#120Earlier quoted context omitted.
There are keysharing protocols; you can do something like 5 sysadmins have a split of the master key such that any 3 of them can access the master account.
I've not needed this, but it's a nice idea. Do you do this with a combination of sudo/PAM|pubkey auth? I can google, but can you push me off in the right direct? Thanks!
I've not been directly involved, so your googling may well be as good as mine; on a quick look you might have to do this manually using ssss (and then each person encrypts their piece with gpg --symmetric or the like).