Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Type in the exact number of machines to proceed

rachelbythebay.com

Post reply on HN
Print view View original

201–210 of 340 posts

Re: Type in the exact number of machines to proceed

#201

Post by matart »

robaato wrote: post #183
harikb wrote: post #62

I have a habit of creating cli tools, which potentially do dangerous things, to default to dry-run mode. For example, instead of the typical `--dry-run` or `-n` option, my scripts instead had a cheesy `--do-it` to be non-dry-run. It is annoying as hell to my colleagues, but saved the day many times.

Bash tip I picked up from observation - always start a potential command with # # rm -rf some_dir Then if you accidentally press return before completing it hasn't happened. When you have reviewed and are sure it is correct, you recall and delete the hash to execute - simples!

Does this work with autocomplete?

Re: Type in the exact number of machines to proceed

#202

Post by __d »

BalinKing wrote: post #175
brlewis wrote: post #143

Earlier quoted context omitted.

>very experienced developers in their 40s I'd say they were experienced developers. Only after accidentally deleting databases were they very experienced developers.

This may not have been their first time, though :-P

Some time ago, it was common in Unix sites to have an NFS filesystem mounted on all machines that contained locally-built binaries to augment those provided by the operating system. At this site, we used a bunch of different platforms: OSF/1, Solaris, Linux, HP/UX, etc. So we had a large filesystem containing the source code, and built binaries for all the different platforms, and this included heaps of things, from Bash upwards.

A colleague of mine accidentally ran rm -rf on this filesystem.

It was taking a loooong time, so he realised and killed it, but not before it had removed a heap of stuff. Because this was something that could be rebuilt, it wasn't backed up, so we had to go through the process of downloading the tarballs, and recompiling everything for all the different platforms. It took a few days to recover most of it, and weeks to completely restore things.

The day after the incident, when he arrived at work, he found his keyboard was missing a few keycaps. It took him a while to realise that there were four gone: 'R', 'M', '-', and 'F' ...

Good times.

Re: Type in the exact number of machines to proceed

#204

Post by arendtio »

robaato wrote: post #183
harikb wrote: post #62

I have a habit of creating cli tools, which potentially do dangerous things, to default to dry-run mode. For example, instead of the typical `--dry-run` or `-n` option, my scripts instead had a cheesy `--do-it` to be non-dry-run. It is annoying as hell to my colleagues, but saved the day many times.

Bash tip I picked up from observation - always start a potential command with # # rm -rf some_dir Then if you accidentally press return before completing it hasn't happened. When you have reviewed and are sure it is correct, you recall and delete the hash to execute - simples!

In my opinion, the option -r should only be allowed as the last parameter. Maybe with the exception of -f. Everything else is just f*ing dangerous.

I mean, I use the # hack sometimes too, but when I don't, I find myself often being afraid of accidentally coming on the enter key.

Re: Type in the exact number of machines to proceed

#205

Post by outworlder »

> 1221425541 machines will be affected

"Do you care? (Y/N)"

Cattle, people. Not pets. Just make sure you don't hit all machines simultaneously and are rolling, instead.

Since the post is talking about automation anyway, assume that any machine that can go down will go down. Ensure that any such disruption will be minimal. Oops, you just killed the production database? Whatever, who cares, it has just failed over anyway (or, for a distributed one, a new node was elected, data started replicating, etc).

If one considers having to SSH to a machine to be an anti-pattern, it's amazing how much crap goes away.

In the more generalized case, where it's not about machines, then it makes more sense. Maybe you are running a query that's going to perform updates across multiple clusters. It still should not be done by hand with direct production access - unless you are in the middle of a declared (and urgent!) incident and everything is on fire. In which case there's a bunch of people watching over your shoulder (or more likely, screen sharing in a conference call).

The same job you have (hopefully) run in QA you should be able to re-target to production. Make the question just be a way to "unlock" your automation - for instance, by not copying credentials or environment information until the proper confirmation has been received. One should still have an escape hatch for when (not IF) things go wrong.

Re: Type in the exact number of machines to proceed

#206

Post by Kerrick »

aqme28 wrote: post #174

Nitpicking > "This might be as simple as printing the number with your locale's version of numerical separators, like "123,456" or "123.456" or "123 456" or whatever else you might use where you are. The trick is then to NOT accept that as input, but instead demand that they remove the separator and jam it in as just digits. " It's easier to just strip non-digit characters than to parse the input for them and respond…

Stripping the non-digit characters would allow "123,456" to validate instead of only accepting "123456" -- which defeats the whole purpose of printing the number with numerical separators (to prevent copy/paste).

Re: Type in the exact number of machines to proceed

#207

Post by Xylakant »

encom wrote: post #166
js2 wrote: post #99

Earlier quoted context omitted.

I learned a technique from a gray beard[0] when I worked as a student sys admin for the CS dept over two decades ago. Whenever typing a destructive command, he'd take his hands off the keyboard and drop them to his side, re-read the command, then put his hands back to press enter. I do this whenever I'm on a production server (which is rare anyway). I use different colored prompts for local and remote shells. [0] Tec…

Re: Beards, color of: Mine started turning grey in my mid 20s. Could be related to me doing the electricians equivalent of deleting production DBs. I've drilled through the comms cable to payment terminals during opening hours. I've run over a copper gas line with a scissor lift. And yes, I've cut live 230V cables with hand tools. That sinking feeling in your stomach you get immediately after doing something bad - it…

> I've drilled through the comms cable to payment terminals during opening hours.

A friend of mine who does fire alarm systems was tasked to install one at a bank branch. He found out the hard way that one of the cables for the safes safety system wasn’t in the place where it should have been according to the plans. Safe’s safety system hosed, bank branch closed for repair.

Re: Type in the exact number of machines to proceed

#208

Post by Lex-2008 »

Gene_Parmesan wrote: post #130
txutxu wrote: post #122

Earlier quoted context omitted.

I worked at a company were someone deleted the production RDS and all the snapshots. Typing the confimation and requesting to delete the snapshots. He had two brosers open, one for development (of cloudformation, etc)... but someone did ask him to change a thing in prod. Both browsers were identical. Only the account in the top right corner did change. Both cloudformation stacks were identical (instance names, etc).…

Firing the person who happened to be at the wheel when a mistake like this occurs never seems like the right choice to me, especially if their performance to-date had otherwise been good. Everybody has off days, or just instances where circumstances misalign in just the wrong way. To pretend otherwise is silly; instead, it's the leader's/team's responsibility to ensure that those sort of off days don't lead to massiv…

> These incidents usually sound simple at the surface level ("I clicked the button in the wrong window") but often hint at deeper, perhaps even organizational, issues.

These words reminded me a story of similar/different "flaps" and "landing gear" controls on a plane - where crashed airplanes were also blamed on pilots first, before a trivial engineering/UI solution was implemented: https://www.endsight.net/blog/what-the-wwii-b17-bomber-can-t...

Re: Type in the exact number of machines to proceed

#209

Post by abrookewood »

jlmorton wrote: post #180
brundolf wrote: post #127

Earlier quoted context omitted.

I've only been in the job field for six years, and yet: My first boss accidentally deleted our QA database, meaning to delete a local copy A later boss accidentally deleted our production database, thinking it was the clone that he had just made (which luckily we still had) Both of them were very experienced developers in their 40s. Nobody is beyond this kind of mistake.

War story time. Long ago, I worked for an interesting company that insisted on running its entire business on Linux desktops, all the way back between 1999-2002. Imagine running StarOffice/OpenOffice, Thunderbird, Netscape Navigator, etc, for your entire business back in 2000, including your executive team, marketing teams, everyone, most of whom had never even heard of Linux before. Anyway, this being Linux, everyon…

That sinking feeling and cold panic when you realise what you've done. God that is horrible.

Re: Type in the exact number of machines to proceed

#210

Post by mamon »

adiM wrote: post #144
roland35 wrote: post #75

Earlier quoted context omitted.

Embarrassingly I once lost a hamburger while still holding it.. I had my arm propped up on a the back of the chair and it was just out of my peripheral vision. Not my smartest moment.

I lost my sunglasses when I was wearing them! We were going to a state park for a hike. It was a 2 hr ride for which I was wearing my sunglasses but forgot. As we came out of the car to start the hike, I spent 5 minutes searching for my sunglasses in my backpack until my friend asked what I was searching for .... Maybe I should be saying "sunglasses on" from now on

Funny, there is a Polish rhyme [1] for children based on the same concept: a person searching the whole house looking for glasses which they were wearing all the time :)

[1] https://blogs.transparent.com/polish/okulary-by-julian-tuwim... (scroll down for english version)

Post reply on HN