31–40 of 129 posts
As I've always said, "You can never protect a system from a stupid person with root". You can limit carnage and mitigate this type of thing, but you can't fully protect against sysadmins doing dumb things (unless you just hire great sysadmins)
So don't give anyone root on an entire data center .
Earlier quoted context omitted.
Sorry - are you telling us you had to reboot all nodes because you swapped a router out? Sounds like you need a network engineer.
And I'm being downvoted for that? Seriously? In 13 years of networking I have never once had to reload machine to help with OSPF or BGP convergence. Good networking architecture and planning should mitigate anything other than a couple of minute outage. No routing change should ever require a reload of a server or end node.
I've been down voted several times for (what I see) as relatively minor remarks. The HN readers are a sensitive bunch...
Earlier quoted context omitted.
> sounds like they're throwing a sysadmin under the bus at least they didn't name the operator in question...
Our internal culture is such that everyone on the team would rather be blamed for something than accuse someone else of doing it. That's shitty, and not something you do to someone. You fix the problem and then you move on. If it makes you happy, blame me - I don't mind.
salt '*' system.reboot
oh shit i meant stag fuckfuckfuckfuck
Earlier quoted context omitted.
So don't give anyone root on an entire data center .
That has its own risks. There might be some catastrophe that need root access on everything to fix, and you can't reach enough people to get it....
salt '*' system.reboot
Looks like the janitor needed somewhere to plug in the vacuum cleaner again...
Earlier quoted context omitted.
As a Devops, I can't justify building any automated way to down or restart all of my systems at once. We've only had to do that to resolve router reconvergence storms when changing out (relatively) major infrastructure pieces, such as our Juniper router.
You don't intentionally build an automated way to take down all your servers at once. You build a way to automatically perform some mundane standard procedure, like propagating a new firewall rule to all your systems at once. Then you accidentally propagate a rule that blocks all inbound ports. Huh, when I tested locally I didn't notice that. Or you build a way to automatically delete timestamped log files more than…
rm -rf logs_ *
rm -rf logs_*Let this be a lesson to linux admins. Re-alias shutdown -r now into something else on production servers. I once took down access to about 6000 servers because I ran the script to decommission servers on our jump box when I got the SSH windows confused.
I once put `shutdown -h now` (halt) instead of `shutdown -r now` (reboot) Once I realized what had happened on the production server I ended up calling OVH (and they were helpful but not immediately acting). It's not a good feeling.
Give it a try on some system that's not critically important :)