Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type in the exact number of machines to proceed

rachelbythebay.com

Post reply on HN
Print view View original

241–250 of 340 posts

Re: Type in the exact number of machines to proceed

#241

Post by xvf22 »

csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

Killed just under 1k access points when they all upgraded on one go. They had no problem erasing the firmware but when they all tried to download the new one at once it killer the service and we ended up with a lot of blank APs. The conformation message for 1 or 1000 APs is unhelpfully "This will overwrite all existing system images. Are you sure Y/N"

Re: Type in the exact number of machines to proceed

#242

Post by rurp »

morty_s wrote: post #65
blantonl wrote: post #21

Earlier quoted context omitted.

Watch and listen to pilots as they complete checklists. They point and callout each item, switch setting, etc.

Came here for this. A: “Passing control” B: “Taking control” A: “You have control” B: “I have control” This is how I remember it (6174, UH-1Y).

Rock climbing is remarkably similar. When a climber begins up a route the standard exchange with their belayer (the person managing the rope and keeping them alive in a fall) goes something like

A: "Belay on?"

B: "Belay on"

A: "Climbing"

B: "Climb on"

Then the climber begins.

It's interesting to me that highly regulated and totally unregulated activities have evolved extremely similar processes. I suppose having your life on the line is a good motivator to follow best practices.

Re: Type in the exact number of machines to proceed

#243

Post by myself248 »

morelisp wrote: post #179
MaxBarraclough wrote: post #13

Earlier quoted context omitted.

Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.

I've done this for several years (also after seeing a video about Japanese railway operations). It doesn't seem to catch on. It's also not perfect; it does not catch mistakes concerning "non-local" state, e.g. configuration files in /etc merging with one in . merging with some command line options. (Personally I try to avoid writing tools with defaults of this sort, but especially Java developers seem have different…

Around industrial machines, I've long held and promoted the view that the machine is _trying_ to kill you, _trying_ to damage itself, _trying_ to ruin the workpiece. Only by outsmarting it at every turn, and having safeguards against every mishap, can you go home at the end of the day.

When something happens despite all that, just step back and realize how much worse it could've been, and how successful your safeguards have been up 'til that point.

Then look carefully at the procedure. Is there something about the naming or structure that could be more clear? Can you think of near-misses that resemble the failure you just experienced? Are you using boobytraps in production? Symlinks and overlay filesystems seem clever in the moment but they're bound to subvert our intuition someday. Perhaps you should get in the habit of always using full absolute paths, for instance.

There's always another gotcha, but if your workflow doesn't look as over-the-top safety-silly as aerospace, you're not doing as much as you could be. (Hint: It's not silly.)

Re: Type in the exact number of machines to proceed

#244

Post by jodrellblank »

luhn wrote: post #81

One of the largest AWS outages to date was caused by a scenario like this. [1] A mistyped commanded removed too many servers from an S3 subsystem, overloading the remaining servers and crashing the subsystem. The failure snowballed until the entire S3 region was down, which then caused issues with dependent services like EBS, ALB, and Lambda. They couldn't even update the status page because that also depended on S3.…

>"overloading the remaining servers and crashing the subsystem. The failure snowballed until"

the entire Eastern Seaboard was without power?

https://youtu.be/XetplHcM7aQ?t=693 (James Burke's Connections, ref. cascading power cut 1965)

Re: Type in the exact number of machines to proceed

#245

Post by jodrellblank »

chrisseaton wrote: post #31
roydivision wrote: post #12

Reminds me of the proposal to keep the nuclear launch codes inside the body of an innocent volunteer, so the President would have to kill the person to get the codes. https://boingboing.net/2015/12/11/proposal-keep-the-nuclear-...

I've never understood this idea. If you believe we should never use nuclear weapons, then don't have them at all. If you believe there is a case where it may be moral and rational to use nuclear weapons, why would you want to put a potential barrier in the way of their use? You could have a situation where everyone was agreed to use them but the president was physically unable to harm the aide to use them. You can kn…

> "If you believe there is a case where it may be moral and rational to use nuclear weapons, why would you want to put a potential barrier in the way of their use?"

Because you think the point where they become moral and rational to use is way way way further than commonly discussed, and you want to put many barriers of many kinds (physical, emotional, logistical) to delay their point of use without completely blocking them.

You could also say that if a person is incapable of doing the hard parts of the job, don't vote them into the position. (Downside of that is that you'll end up voting someone who doesn't mind killing someone in cold blood while expecting that to be a filter that brings more empathy to the position).

Re: Type in the exact number of machines to proceed

#246

Post by grossvogel »

morelisp wrote: post #178
js2 wrote: post #99

Earlier quoted context omitted.

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Tec…

A similar tip I picked up long ago: If you're typing a dangerous command, first type a `#` (or `--` if it's SQL, etc.), then the command. Then read it. Then go back to the start of the line and remove the comment and run it.

I always do destructive SQL commands in two steps: first run a select using the WHERE clause you intend to use and verify which records will be affected, then hit the up arrow and edit the beginning of the query leaving the WHERE intact.

I also like adding redundant conditions to the WHERE so a typo in any single one of them won't sink me.

Re: Type in the exact number of machines to proceed

#247

Post by coredog64 »

jlmorton wrote: post #180
brundolf wrote: post #127

Earlier quoted context omitted.

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

War story time. Long ago, I worked for an interesting company that insisted on running its entire business on Linux desktops, all the way back between 1999-2002. Imagine running StarOffice/OpenOffice, Thunderbird, Netscape Navigator, etc, for your entire business back in 2000, including your executive team, marketing teams, everyone, most of whom had never even heard of Linux before. Anyway, this being Linux, everyon…

Similar story, except we were using an NFS appliance that took hourly snapshots. As soon as we figured out what was happening, we had the storage team save off the latest snapshot. It was 1TB of data (a lot for the time) and took a week for us to restore.

Re: Type in the exact number of machines to proceed

#248

Post by _asummers »

Gene_Parmesan wrote: post #130
txutxu wrote: post #122

Earlier quoted context omitted.

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massiv…

There's a really good book describing this phenomenon called Behind Human Error. It speaks of "first stories" and "second stories" and how in analysis of incidents, it is all too common to stop at the first story and chalk it up to human error, when the system itself allowed it to take place.

Re: Type in the exact number of machines to proceed

#249

Post by jeffbee »

jaclaz wrote: post #46
notacoward wrote: post #36

Earlier quoted context omitted.

At least Facebook (where OP worked), Amazon, Google, and Microsoft. Probably Netflix, maybe Apple. There might be a couple more, but no more than that because we've already accounted for a pretty high percentage of worldwide shipments for servers, disks, etc. Fun fact: when you're that big, your demand creates its own inflation and you have to consider that in projections.

Thanks, so a handful at most, and the "usual" ones, I always thought that those companies keep their machines connected in (redundant) "sets" and that a command affecting all of them was more a case for "never" rather than "once in a while".

Google, at least, has a thing that is supposed to prevent widespread disruption at the machine level, called the "Safe Removal Service"[1]. This is a good idea that in practice isn't perfect. If you write a tool that does not consult SRS, or your service doesn't declare a SRS policy, there can be surprises.

A particular outage that I will never forget took out Gmail delivery worldwide in an instant, because the change was not expected to be disruptive and therefore did not integrate with SRS. As it turned out the change disabled the machines where it was applied, and the process of selecting a subset of machines to canary the change was not independent of the way in which Gmail assigns services to machines, so in the space of a few seconds they created a global outage.

https://twitter.com/bgrant0607/status/1134536670504554496

Re: Type in the exact number of machines to proceed

#250

Post by acdha »

csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

Back when I shelled into servers more, I really liked having my deployment put the environment in the prompt and set a red background on production for similar reasons. It only takes a small change to jar you out of habit.
Post reply on HN