Type in the exact number of machines to proceed • Hacker News Board

Re: Type in the exact number of machines to proceed

#291

Post by Huggernaut » Wed, Oct 28, 2020, 8:02 AM UTC

Gene_Parmesan wrote: post #130

txutxu wrote: post #122

Earlier quoted context omitted.

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massiv…

Nickolas Means has an absolutely wonderful set of talks on themes like this. Particularly relevant here I think, is his talk: "Who Destroyed Three Mile Island?" - which goes through the events that occurred at the nuclear power plant, the systemic problems, and how to find the "second stories" of why failures occurred.

https://www.youtube.com/watch?v=1xQeXOz0Ncs

Re: Type in the exact number of machines to proceed

#292

Post by dirkt » Wed, Oct 28, 2020, 8:06 AM UTC

myself248 wrote: post #227

csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

Ever since hearing about point-and-call, I've started using it in the kitchen when turning on the stove. I used to destroy one or two pans a year by turning on the wrong burner, but it's now been about a year and a half and I haven't screwed it up yet. The knobs are labeled with a terrible little glyph meant to indicate which is which, and I've supplemented this with plain-english Brady labels "front left", "front ri…

Not sure how it is in other countries, but don't the knobs when going left-to-right always correspond clockwise to the burners, starting at the lower left? And the oven knob is to the right?

I've never seen a different arrangement.

Re: Type in the exact number of machines to proceed

#293

Post by chrisseaton » Wed, Oct 28, 2020, 8:19 AM UTC

Aeolun wrote: post #233

chrisseaton wrote: post #101

Earlier quoted context omitted.

> It is about forcing the president to look somebody in the eye before they kill them. Right, but can you understand that 'the President being able to look somebody in the eye before they killing them' is not a requisite for 'the employment of nuclear weapons being justified'? We require the president to be able to do B before they can do A. But what if A is the right thing to do but the President is not able to do B…

Doing A cannot be the right thing to do if you think doing B is still impossible. If you cannot kill your friend to kill a few hundreds of thousands more, how can it possibly be justified? I just struggle to come up with a scenario where that is the case. Of course I’m of the school that thinks firing nuclear weapons is never a good idea.

But ‘know you should’ is not the same as ‘can physically bring yourself to do it’.

Re: Type in the exact number of machines to proceed

#294

Post by ajb » Wed, Oct 28, 2020, 8:22 AM UTC

giantDinosaur wrote: post #288

myself248 wrote: post #227

Earlier quoted context omitted.

Ever since hearing about point-and-call, I've started using it in the kitchen when turning on the stove. I used to destroy one or two pans a year by turning on the wrong burner, but it's now been about a year and a half and I haven't screwed it up yet. The knobs are labeled with a terrible little glyph meant to indicate which is which, and I've supplemented this with plain-english Brady labels "front left", "front ri…

I'm curious how exactly you managed to destroy pans. I've never destroyed a pan in my life, and take no particular precautions - is this a common thing? Is this more common with non-stick stuff or something?

Not the op, but non stick pans will burn if the pan is heated while empty.

Re: Type in the exact number of machines to proceed

#295

Post by Fradow » Wed, Oct 28, 2020, 8:51 AM UTC

morelisp wrote: post #178

js2 wrote: post #99

Earlier quoted context omitted.

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Tec…

A similar tip I picked up long ago: If you're typing a dangerous command, first type a `#` (or `--` if it's SQL, etc.), then the command. Then read it. Then go back to the start of the line and remove the comment and run it.

I use an alternate version on SQL: when running any modification on any kind of sensible database (which is a bad practice in itself, obviously, but sometimes you don't have a choice), always type in the WHERE clause before the table name (added bonus: do a SELECT first with that clause to see what you are modifying).

That way, if you accidentally send it, the command fails and nothing happens.

Re: Type in the exact number of machines to proceed

#296

Post by edanm » Wed, Oct 28, 2020, 8:54 AM UTC

dgritsko wrote: post #2

Similar idea as GitHub's "type the exact name of this repository if you want to delete it" confirmation dialog. Maybe that's really what you want to do, but in case that's not actually what you meant to do, having a few extra hoops to jump through seems like a good idea.

That's what I thought of immediately as well! I've seen that pattern in a few other places too, and I always think it's a really good UX choice.

Re: Type in the exact number of machines to proceed

#297

Post by b6z » Wed, Oct 28, 2020, 9:28 AM UTC

remram wrote: post #135

A similar system is molly-guard [1], which replaces the reboot/halt/poweroff/... commands with scripts that make you type in the name of the machine before proceeding. Avoids shutting down the wrong machine because you forgot where you SSH'd. [1]: https://manpages.debian.org/buster/molly-guard/molly-guard.8...

Many years ago, I made that mistake two or three times, rebooting the wrong machine. Since then, I use molly-guard on all my remote machines. Never happened again.

Re: Type in the exact number of machines to proceed

#298

Post by davedx » Wed, Oct 28, 2020, 9:28 AM UTC

brundolf wrote: post #127

csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

Yup. Senior dev here, my own devops config screw up wiped out all production sales order data earlier this year. Had to restore from multiple backups, took a while. Stressful experience.

Consider network partitioning so dev/test/accept just has 0 contact with prod.

Re: Type in the exact number of machines to proceed

#299

Post by YeGoblynQueenne » Wed, Oct 28, 2020, 9:36 AM UTC

kbenson wrote: post #116

YeGoblynQueenne wrote: post #102

Earlier quoted context omitted.

>> I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. Yeah, ouch. More ouch if it's the other way around- you delete the test database and it's not the test database. (long story)

> you delete the test database and it's not the test database. > (long story) I think you can skip the long story, as most of us can tell a story similar in theme if not specifics (and sometimes, probably some similar specifics too). ;) With great power comes great responsibility (to not completely screw stuff up because you were on autopilot for a second...)

Indeed. God damn muscle memory.

Re: Type in the exact number of machines to proceed

#300

Post by YeGoblynQueenne » Wed, Oct 28, 2020, 9:40 AM UTC

txutxu wrote: post #122

throwaway894345 wrote: post #107

Earlier quoted context omitted.

I worked at a company where someone deleted the production database by accident and the snapshot mechanism hadn't been working AND the alerting for the snapshot mechanism was also broken. Fortunately someone had taken a snapshot manually some weeks prior and they were able to restore from that and lose relatively little data (it was a startup, so one database was a big deal, but weeks worth of data was not such a big…

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

That's very similar to what happened to me - except I didn't delete any backups, thank the Great Old Ones. And I didn't get fired.

Basically, I had a habit of starting a new SQL Server Management Studio instance in its own window for each database I was working on. At some point this struck me as wasteful, for some reason, so I closed all my windows and opened all the databases in one window. Then sometime after that I went to delete the test database as a routine maintainance task, but of course I was used to clicking the database at the top of the left pane in SSMS, which was the test database when it was the only database in a window... but now happened to be the production database. Then five minutes later I got a call from the client company that used our system, to ask me if there was any maintainance going on because everyone's client had just crashed.

The horror when I realised.

It was educational, though. I don't think I'll make that particular mistake ever again. And my bosses were ace to be fair, probably because I worked my ass off to correct the mess that ensued.