Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Joyent us-east-1 rebooted due to operator error

help.joyent.com

Post reply on HN
Print view View original

11–20 of 129 posts

Re: Joyent us-east-1 rebooted due to operator error

#12

Post by stephengillie »

jameshart wrote: post #9

DevOps means being able to take out an entire datacenter with a single keysstroke...

As a Devops, I can't justify building any automated way to down or restart all of my systems at once. We've only had to do that to resolve router reconvergence storms when changing out (relatively) major infrastructure pieces, such as our Juniper router.

Re: Joyent us-east-1 rebooted due to operator error

#13

Post by _hnwo »

shiftpgdn wrote: post #7

Let this be a lesson to linux admins. Re-alias shutdown -r now into something else on production servers. I once took down access to about 6000 servers because I ran the script to decommission servers on our jump box when I got the SSH windows confused.

Might I suggest molly-guard: https://packages.debian.org/unstable/admin/molly-guard

Re: Joyent us-east-1 rebooted due to operator error

#14

Post by wmf »

SEJeff wrote: post #3

As I've always said, "You can never protect a system from a stupid person with root". You can limit carnage and mitigate this type of thing, but you can't fully protect against sysadmins doing dumb things (unless you just hire great sysadmins)

So don't give anyone root on an entire data center.

Re: Joyent us-east-1 rebooted due to operator error

#16

Post by icebraining »

cordite wrote: post #10
shiftpgdn wrote: post #7

Let this be a lesson to linux admins. Re-alias shutdown -r now into something else on production servers. I once took down access to about 6000 servers because I ran the script to decommission servers on our jump box when I got the SSH windows confused.

I once put `shutdown -h now` (halt) instead of `shutdown -r now` (reboot) Once I realized what had happened on the production server I ended up calling OVH (and they were helpful but not immediately acting). It's not a good feeling.

I tend to use /sbin/reboot instead, it amounts to the same (calls shutdown), but it's harder to get it mixed up.

Re: Joyent us-east-1 rebooted due to operator error

#17

Post by akerl_ »

wmf wrote: post #14
SEJeff wrote: post #3

As I've always said, "You can never protect a system from a stupid person with root". You can limit carnage and mitigate this type of thing, but you can't fully protect against sysadmins doing dumb things (unless you just hire great sysadmins)

So don't give anyone root on an entire data center .

Is this like Captain Planet? It's a bit exceptional to divide access servers of similar type between administrators such that individuals have full access to a portion of the fleet. Do they meet up and put their rings together to roll out updates? What if one of them goes on vacation?

Re: Joyent us-east-1 rebooted due to operator error

#18

Post by mikeash »

wmf wrote: post #14
SEJeff wrote: post #3

As I've always said, "You can never protect a system from a stupid person with root". You can limit carnage and mitigate this type of thing, but you can't fully protect against sysadmins doing dumb things (unless you just hire great sysadmins)

So don't give anyone root on an entire data center .

That has its own risks. There might be some catastrophe that need root access on everything to fix, and you can't reach enough people to get it....
Post reply on HN