Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type in the exact number of machines to proceed

rachelbythebay.com

Post reply on HN
Print view View original

121–130 of 340 posts

Re: Type in the exact number of machines to proceed

#121

Post by GauntletWizard »

wruza wrote: post #110
apozem wrote: post #57

Earlier quoted context omitted.

In high school, I drove a 1993 Toyota Tercel. It was a functional, reliable car, but it had no keyfob to lock the doors remotely. Getting out of your car, pressing the lock button on the inside of the driver's side door, and shutting the door are all routine, boring actions that make it easy to forget your keys inside the car. The keys can go in all kinds of places as you climb out of the car - jacket pocket, pants p…

I'm always using the "phone keys cigarettes money" mantra together with patting on my pockets before opening any outside door.

I wake up in the mornings with "Shit Shower Shave" and leave the house with "Wallet Watch Testicles Spectacles". Simple mnemonics work, doubly so if you actually say them out loud and check them each off.

Re: Type in the exact number of machines to proceed

#122

Post by txutxu »

throwaway894345 wrote: post #107
YeGoblynQueenne wrote: post #102

Earlier quoted context omitted.

>> I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. Yeah, ouch. More ouch if it's the other way around- you delete the test database and it's not the test database. (long story)

I worked at a company where someone deleted the production database by accident and the snapshot mechanism hadn't been working AND the alerting for the snapshot mechanism was also broken. Fortunately someone had taken a snapshot manually some weeks prior and they were able to restore from that and lose relatively little data (it was a startup, so one database was a big deal, but weeks worth of data was not such a big…

I worked at a company were someone deleted the production RDS and all the snapshots.

Typing the confimation and requesting to delete the snapshots.

He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod.

Both browsers were identical. Only the account in the top right corner did change.

Both cloudformation stacks were identical (instance names, etc).

He had been all the morning launching and deleting the dev environment.

Team mates were joking loud around his table before the moment it did happen.

Sadly, he got fired (the company was proud of it's cost savy choices, didn't have other backups than a few days of snapshots, probably CTO choice).

Re: Type in the exact number of machines to proceed

#123

Post by rachelbythebay »

abnry wrote: post #88
jaclaz wrote: post #25

Side question. How many/which companies have more than one million Linux machines?

The number blew me away. But does she mean in one location or VMs? One million is a lot no matter how you slice it.

How do you define one location? If it's like, a contiguous plat of land with a bunch of buildings, each containing suites, and each of those containing clusters... then these days, yeah, that's probably not too much of a stretch.

And yeah, physical machines, not VMs. Sometimes they're blades, sometimes they're sleds, but I mean real hardware made out of metal that you can pick up and use to defend the datacenter if you have to.

(Although, honestly, I was talking about global counts in the million+ range when I wrote it since it was referencing the past, but by now, a region with a million+ is not far-fetched.)

Re: Type in the exact number of machines to proceed

#124

Post by _ikke_ »

xaedes wrote: post #95
leetcrew wrote: post #85

Earlier quoted context omitted.

works great until some asshole puts alias harikb_script='harikb_script --do-it' in their .bashrc to eliminate this annoying step.

This would help a bit: Don't accept the "--do-it" as first parameter, make it obligatory to be the last.

  my_command() {
      command my_command "$@" --do-it
  }

Re: Type in the exact number of machines to proceed

#125

Post by uyt »

csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

This is true for NYC subways too! https://www.youtube.com/watch?v=i9jIsxQNz0M

Re: Type in the exact number of machines to proceed

#126

Post by rachelbythebay »

bronco21016 wrote: post #120

It amazes me that something like this can be done by a single person. In aviation any time input is given to the machine, it's entered by one human (typically pilot flying) and then verified by the other human (typically pilot monitoring) before being committed to or executed. For example... when a new altitude is assigned by ATC, say FL300, the pilot flying will spin it in the selector window and keep his hand or fi…

Best practice for using the "weaponized" version of the tool when you had powers to actually hit all of them at once was to paste the command into IRC and get some of your fellow peeps to eyeball it and make sure it was sane.

team: hey, sanity check this please: hsh -A "dumb_thing && other_thing --foo --bar" shipit

[ I type the command ]

ok, running as job 1234

The last part was a courtesy done so that they could watch the progress of it too without having to dig to find my request. It also meant they could kill it easily if something went wrong and they couldn't raise me for some reason.

Tools like this are best used outside the solo realm.

Re: Type in the exact number of machines to proceed

#127

Post by brundolf »

csmattryder wrote: post #8

I've seen this called "pointing and calling" [1], Japan's train drivers use the technique to force themselves to perform actions and take notice of the current environment. I personally took it to heart, it's a good system for forcing a cache miss in the brain - make sure you're on "database production" or "database localhost" etc. [1] https://en.wikipedia.org/wiki/Pointing_and_calling

I've only been in the job field for six years, and yet:

My first boss accidentally deleted our QA database, meaning to delete a local copy

A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had)

Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

Re: Type in the exact number of machines to proceed

#128

Post by actuallyalys »

leetcrew wrote: post #85
harikb wrote: post #62

I have a habit of creating cli tools, which potentially do dangerous things, to default to dry-run mode. For example, instead of the typical `--dry-run` or `-n` option, my scripts instead had a cheesy `--do-it` to be non-dry-run. It is annoying as hell to my colleagues, but saved the day many times.

works great until some asshole puts alias harikb_script='harikb_script --do-it' in their .bashrc to eliminate this annoying step.

I suspect someone who'd do that isn't going to take that or other precautions seriously regardless of it being aliased. It's still a problem that they're circumventing it, but I think you have a larger problem if someone with that mindset has access to production.

Re: Type in the exact number of machines to proceed

#129

Post by brundolf »

js2 wrote: post #99
MaxBarraclough wrote: post #13

Earlier quoted context omitted.

Worth mentioning that, assuming the single study on the matter can be believed, the pointing and calling method is extremely effective in reducing the incidence of silly mistakes (that is, mistakes made in simple routine tasks, by competent individuals). Unfortunately, it strikes many as looking rather silly, so it hasn't been widely adopted.

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Tec…

Different-colored prompts for different machines is a great thing to do (I've been doing it for years), and very easy to implement

Re: Type in the exact number of machines to proceed

#130

Post by Gene_Parmesan »

txutxu wrote: post #122
throwaway894345 wrote: post #107

Earlier quoted context omitted.

I worked at a company where someone deleted the production database by accident and the snapshot mechanism hadn't been working AND the alerting for the snapshot mechanism was also broken. Fortunately someone had taken a snapshot manually some weeks prior and they were able to restore from that and lose relatively little data (it was a startup, so one database was a big deal, but weeks worth of data was not such a big…

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good.

Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massive losses via redundancy & the sort of measures we're talking about here & in the OP. Firing somebody in these circumstances just acts to severely reduce morale, since we all secretly know in our hearts that it very easily could have been us.

Firing in this case just seems retributive. It's not going to bring the lost data back, and you've just eliminated the very person who could have told you most about the chain of events leading to the incident in question to help you guard against it in the future. These incidents usually sound simple at the surface level ("I clicked the button in the wrong window") but often hint at deeper, perhaps even organizational, issues. A lack of team focus on reliability/quality, a lack of communication or trust about decisions made (or not made) by higher ups, or so on.

And they are probably the single least likely person to cause a similar incident again -- that person will now likely be double and triple checking their commands for eternity.

Post reply on HN