81–90 of 129 posts
Earlier quoted context omitted.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.
really? what can cost that much ?
Earlier quoted context omitted.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.
That sounds so awful. I can't imagine living the rest of my life knowing that I had been a net negative in the world. All of my life's earnings would just be a partial restitution of that one second of destruction.
Earlier quoted context omitted.
That has its own risks. There might be some catastrophe that need root access on everything to fix, and you can't reach enough people to get it....
Then put the keys to datacenter-wide root somewhere safe (with a manual-ish process to access and use them), but out of the way and with alarms on it (the same alarms that you'd use in the absolute worst situation possible). Make sure anyone using it will be shamed if they don't absolutely have to.
It's also a cautionary tale about testing your DR occasionally.
Earlier quoted context omitted.
I feel bad for the person who made the mistake. Even though its obviously a systemic problem, and highly unlikely to be an act of negligence, Im sure he/she doesnt feel too hot right now.
My thoughts exactly. Poor fella. I've seen worse though. A newish officer spilled his morning coffee into the circuitry of a device worth over 10 zeros. Immediately short circuited.
Why, yes...a couple of times...before a perspex arch was less-than-hastily fixed over the button..
Earlier quoted context omitted.
Systems engineers, software engineers, architects, whatever. We're all in the same gang. My point is that the problem in this case is likely the system's design, not one engineer's typing abilities.
This comes down to operational philosophy, in the end. The point you're dancing around is whether the system should permit grave actions that don't make any sense when you're designing the system. By the rules, every single system on a commercial aircraft has a circuit breaker. Pilots make the "what if X catches on fire?" case, which is actually pretty compelling. However, that also means there are several switches o…
[0] http://www.joyent.com/products/compute-service/features/linu...
Earlier quoted context omitted.
I once put `shutdown -h now` (halt) instead of `shutdown -r now` (reboot) Once I realized what had happened on the production server I ended up calling OVH (and they were helpful but not immediately acting). It's not a good feeling.
This happened to me once; I don't know if this works on all linux distros but if you quickly follow a halt/shutdown with a "sudo init 6"(reboot) before your ssh-session gets SIGTERMed/KILLed, the box comes back up. This at least worked on some Ubuntu version a few years back. Give it a try on some system that's not critically important :)
Earlier quoted context omitted.
This comes down to operational philosophy, in the end. The point you're dancing around is whether the system should permit grave actions that don't make any sense when you're designing the system. By the rules, every single system on a commercial aircraft has a circuit breaker. Pilots make the "what if X catches on fire?" case, which is actually pretty compelling. However, that also means there are several switches o…
I tend to agree with you, with the caveat that you can't have this philosophy and sell your customers 99.999% uptime[0]. [0] http://www.joyent.com/products/compute-service/features/linu...
Earlier quoted context omitted.
You don't intentionally build an automated way to take down all your servers at once. You build a way to automatically perform some mundane standard procedure, like propagating a new firewall rule to all your systems at once. Then you accidentally propagate a rule that blocks all inbound ports. Huh, when I tested locally I didn't notice that. Or you build a way to automatically delete timestamped log files more than…
Once I typed rm -rf logs_ * instead of rm -rf logs_*
All was well until a scheduled maintenance restart a few weeks later and we (eventually) discovered that /boot and /bin were AWOL.
Edit: He had access to the root account to maintain the accounts app (not my call)