Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type in the exact number of machines to proceed

rachelbythebay.com

Post reply on HN
Print view View original

161–170 of 340 posts

Re: Type in the exact number of machines to proceed

#161

Post by nitrogen »

mehrdadn wrote: post #159
greedo wrote: post #149

Earlier quoted context omitted.

Sometimes. And sometimes they make the same mistake over and over. We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours. Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luc…

3 mistakes in... >2 years? I feel like it's really hard to tell if the problem is really the person at that point. Have you had others perform the same job for a similar duration to see if they avoid the same mistakes?

If you made a list of every mistake each person makes in 2-3 years, and omitted all other detail, pretty much everybody would look like a terrible person. Context, frequency, etc. all matter.

If particular systems or people are seeing a high frequency of mistakes, maybe the system design is at fault, not just the person. Obviously it's hard to do in practice, but the ideal is to design systems that are mistake proof.

Re: Type in the exact number of machines to proceed

#162

Post by nailer »

csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

I do things like

  const HARD_CODE_TEST_DATABASE_FOR_SAFETY = 'unit-testing'

  destroyDatabase(HARD_CODE_TEST_DATABASE_FOR_SAFETY)
1. Avoid silly terms our industry should have ditched years ago, like 'drop'

2. Making sure that nobody will ever change HARD_CODE_TEST_DATABASE_FOR_SAFETY because they thought it should 'always be the active database' or whatever.

Re: Type in the exact number of machines to proceed

#163

Post by the_af »

chrisseaton wrote: post #79
shuntress wrote: post #66

Earlier quoted context omitted.

>The president's ability to physically cut someone open is not relevant to whether it's a good idea to use nuclear weapons or not I'm sure if the president was physically incapable of wielding a knife, she would have someone on hand to do that for her.

I think the whole point of the 'rule' is that they have to do it themselves.

I think it would work equally well if the president had two aides and had to order one to butcher the other, in front of her eyes, in order to launch a nuclear strike.

Regardless of the exact details, I think the point of this thought experiment is that for a head of state, the decision to launch a massive attack that will cause hundreds of thousands of casualties can feel a little abstract. "Bombing a city" can seem abstract, even if the president understands this means killing children. Understanding is quite different from feeling. However, if the act of ordering a bombing raid on a city involved physically murdering a child, it would definitely feel more immediate and less abstract.

Your point stands, of course. But the part about removing the abstractness of the act seems relevant when ordering people killed.

Re: Type in the exact number of machines to proceed

#164

Post by nitrogen »

ahoka wrote: post #141
txutxu wrote: post #122

Earlier quoted context omitted.

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

"Both cloudformation stacks were identical (instance names, etc)." This is why it's a good practice to include the environment name in the resource names when it makes sense. Even better, don't append the env name, but use it as a prefix, like ProdCustomerDb instead of CustomerDbProd. I also like to change the theme to dark mode in the production environments as most management UIs support this. One other neat trick…

One other neat trick is to color code PS1 in your Linux instances, like red for prod, green for dev.

This is definitely a nice one to add. Though I did work with someone once who believed that all servers should be 100% vanilla and reverted my environment colors.

In container-only shops with no ssh, this is less of an issue, and instead you rely on having different permissions and automations for different environments.

Re: Type in the exact number of machines to proceed

#165

Post by Sharlin »

staunch wrote: post #29
blantonl wrote: post #21

Earlier quoted context omitted.

Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.

And pilots will even callout that their action had the desired effect: "Flaps up selected" "Flaps are indicating up" There's a lot to learn from the way airplanes are engineered and operated.

And it should be kept in mind that almost none of those procedures were intuitively obvious things to do. As the saying goes, safety standards are written in blood.

Re: Type in the exact number of machines to proceed

#166

Post by encom »

js2 wrote: post #99
MaxBarraclough wrote: post #13

Earlier quoted context omitted.

Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Tec…

Re: Beards, color of:

Mine started turning grey in my mid 20s.

Could be related to me doing the electricians equivalent of deleting production DBs. I've drilled through the comms cable to payment terminals during opening hours. I've run over a copper gas line with a scissor lift. And yes, I've cut live 230V cables with hand tools.

That sinking feeling in your stomach you get immediately after doing something bad - it's universal across professions.

Thankfully, I've never fucked anything major up, and I've had my hands in hospitals, power plants, ISP fiber backbones, police stations and whatnot.

Re: Type in the exact number of machines to proceed

#167

Post by encom »

wruza wrote: post #110
apozem wrote: post #57

Earlier quoted context omitted.

In high school, I drove a 1993 Toyota Tercel. It was a functional, reliable car, but it had no keyfob to lock the doors remotely. Getting out of your car, pressing the lock button on the inside of the driver's side door, and shutting the door are all routine, boring actions that make it easy to forget your keys inside the car. The keys can go in all kinds of places as you climb out of the car - jacket pocket, pants p…

I'm always using the "phone keys cigarettes money" mantra together with patting on my pockets before opening any outside door.

I do that exact some thing, and I haven't smoked in 3 years. The downside is that if I'm supposed to remember to bring something, in addition to those 3 things, I'm extremely likely to forget it. If it's super duper important, I tie it to the door handle.

Re: Type in the exact number of machines to proceed

#168

Post by woliveirajr »

Once I had to deal with some software-RAID in Linux (mdadm it is), around 2007. There was some -force option that would just print information explaining what it would do and, to perform the real action, you needed to type another flag (that should never be revealed).

Edit: added name of software

Re: Type in the exact number of machines to proceed

#169

Post by the_af »

mikewarot wrote: post #56
gumby wrote: post #37

Earlier quoted context omitted.

> I've never understood this idea. It's an attempt to make an abstraction concrete. Think of it as the trolley problem in real life. Stalin is famously supposed to have said, "one death is a tragedy, 100,000 is a statistic". Cynical or not it is how humans think. > If you believe we should never use nuclear weapons, then don't have them at all. Strategic game theory and Mutual Assured Destruction depend on the possib…

Trolley Problems are themselves a bad idea... the Kobayashi Maru is a similar exercise. I, like Kirk, don't believe that there are situations that can't be worked around if there is time to think, and resources to act.

Isn't the Trolley problem a situation that is, by definition, time sensitive? If you had more time to think and resources to act, it wouldn't be a Trolley Problem.

If the answer to launch-nukes-by-cutting-a-human-aide is "well, I need more time to think" then maybe that's a good outcome?

Re: Type in the exact number of machines to proceed

#170

Post by duskwuff »

Ayesh wrote: post #53

I have an old laptop with a dead battery, and for a BIOS upgrade, it prevents me from updating without 50% battery. I have to type "danger" to bypass this restriction, and I thought it was pretty cool. Another good UI pattern is in Firefox, that it disables the Run button on downloads for a few seconds.

Disabling the "run" button for a few seconds was actually done to mitigate another risk -- sites cueing the user to click in a particular location, then triggering the confirmation dialog with the "run" button right where the user was about to click.
Post reply on HN